2020-06-19.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### past\n",
    "- notebook \"2020-06-14.ipynb\", tested the capability to continue fitting `clf.fit(X[part], y_enc[part], xgb_model=prev_model)` and saving `clf.save_model(prev_model)` a model in a loop, over parts of a training set. \n",
    "- however, the `predict_proba()` step on the test set at each iteration, was taking a very long time. Somtimes half an hour or 40 minutes. \n",
    "- In notebook \"2020-06-16.ipynb\" , I switrched to evaluating on that test set every epoch instead of every batch. Still, this finished `14` epochs in `13` hours and I even cut my data set by `60x` . That also appears to have deteriorated the test accuracy from `acc=0.12` in \"2020-06-14.ipynb\" to `acc=0.021` in \"2020-06-16.ipynb\". Again the train and test sets were cut by `60x` so that definitely changed things. But also the `14` epochs did not appear to be helping. \n",
    "\n",
    "\n",
    "#### other notes\n",
    "- feels like better balanced datasets are important . Perhaps otherwise the time to use them is maybe excessively long.\n",
    "- predict proba is slow ( as per my [2020-06-14 logs here](https://github.com/namoopsoo/learn-citibike/blob/2020-revisit/notes/2020-06-14.md#output-logs) )\n",
    "- I still feel like `100` iterations feels like a minimum of sorts to aim for.\n",
    "- got to re-do 3 or 5 fold cross validation now that weird log loss is good.\n",
    "- If I do the non-sklearn xgboost, I can take advantage of that dataset caching, which I think saved on memory.\n",
    "- So perhaps one goal is to try that non-sklearn xgboost, w/ `100` iterations, but using at least those parameters from bundle `'/opt/program/artifacts/2020-06-11T041641Z/bundle.joblib'` which appeared to be a good start..\n",
    "\n",
    "```\n",
    "\n",
    "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n",
    "               colsample_bynode=1, colsample_bytree=1, gamma=0,\n",
    "               learning_rate=0.1, max_delta_step=0, max_depth=3,\n",
    "               min_child_weight=1, missing=nan, n_estimators=100, n_jobs=1,\n",
    "               nthread=None, objective='multi:softprob', random_state=0,\n",
    "               reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n",
    "               silent=None, subsample=1, verbosity=1),\n",
    "               \n",
    "```\n",
    "\n",
    "#### Are those iterations \"built in\" ? \n",
    "- If I am \"manually\" iterating then am I actually doing 100*100 = 10,000 iterations?! \n",
    "- Perhaps I can check by using the verbose mode\n",
    "\n",
    "#### Memory consumed by other notebooks\n",
    "- also important note that I forgot that other notebooks may have been sitting on memory, slowing down the notebook where I was actually running work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "COMMAND\r\n",
      "/opt/conda/bin/python -m ipykernel_launcher -f /root/.local/share/jupyter/runtime/kernel-5732db1a-d484-4a58-9d67-de6ef5ac721b.json\r\n"
     ]
    }
   ],
   "source": [
    "#!ps -p 689 -o pid,ppid,pmem,rss,res\n",
    "!foo=$(ps -p 689 -o command); echo \"$foo\"\n",
    "#!ps -p 689 -o command\n",
    "# 727\n",
    "# COMMAND /opt/conda/bin/python -m ipykernel_launcher -f /root/.local/share/jupyter/runtime/kernel-11cd26cf-5715-4e84-abd4-ee49b0b06ed9.json\n",
    "\n",
    "# 689\n",
    "# COMMAND /opt/conda/bin/python -m ipykernel_launcher -f /root/.local/share/jupyter/runtime/kernel-5732db1a-d484-4a58-9d67-de6ef5ac721b.json\n",
    "#!cat /root/.local/share/jupyter/runtime/kernel-5732db1a-d484-4a58-9d67-de6ef5ac721b.json\n",
    "#os.getpid()\n",
    "\n",
    "# 667 , the parent\n",
    "# COMMAND /opt/conda/bin/python /opt/conda/bin/jupyter-notebook --ip 0.0.0.0 --port 8889 --no-browser --allow-root\n",
    "#!foo=$(ps auxww); echo \"$foo\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[?1h\u001b=\u001b[H\u001b[2J\u001b[mtop - 12:25:39 up 4 days,  6:45,  0 users,  load average: 0.00, 0.02, 0.00\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "Tasks:\u001b[m\u001b[m\u001b[1m  10 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m   1 \u001b[m\u001b[mrunning,\u001b[m\u001b[m\u001b[1m   9 \u001b[m\u001b[msleeping,\u001b[m\u001b[m\u001b[1m   0 \u001b[m\u001b[mstopped,\u001b[m\u001b[m\u001b[1m   0 \u001b[m\u001b[mzombie\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "%Cpu(s):\u001b[m\u001b[m\u001b[1m  6.7 \u001b[m\u001b[mus,\u001b[m\u001b[m\u001b[1m  0.7 \u001b[m\u001b[msy,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mni,\u001b[m\u001b[m\u001b[1m 92.3 \u001b[m\u001b[mid,\u001b[m\u001b[m\u001b[1m  0.2 \u001b[m\u001b[mwa,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mhi,\u001b[m\u001b[m\u001b[1m  0.1 \u001b[m\u001b[msi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mst\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "KiB Mem :\u001b[m\u001b[m\u001b[1m  2046844 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m    82660 \u001b[m\u001b[mfree,\u001b[m\u001b[m\u001b[1m  1855176 \u001b[m\u001b[mused,\u001b[m\u001b[m\u001b[1m   109008 \u001b[m\u001b[mbuff/cache\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "KiB Swap:\u001b[m\u001b[m\u001b[1m  1048572 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m   695520 \u001b[m\u001b[mfree,\u001b[m\u001b[m\u001b[1m   353052 \u001b[m\u001b[mused.\u001b[m\u001b[m\u001b[1m    50424 \u001b[m\u001b[mavail Mem \u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\u001b[K\n",
      "\u001b[7m  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m    1 root      20   0   18236    204    100 S   0.0  0.0   0:00.11 bash        \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  667 root      20   0  999296  56116   5060 S   0.0  2.7   0:21.62 jupyter-no+ \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  673 root      20   0  555440   8460   3128 S   0.0  0.4   0:24.88 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  689 root      20   0 2521820 1.492g  18052 S   0.0 76.4 760:54.45 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  711 root      20   0  555292  44516   6204 S   0.0  2.2   0:15.04 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  727 root      20   0  555292  44656   6280 S   0.0  2.2   0:15.52 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  743 root      20   0  555292  44644   6316 S   0.0  2.2   0:14.36 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  759 root      20   0  562420  53140  13436 S   0.0  2.6   0:02.17 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  783 root      20   0    4500    672    600 S   0.0  0.0   0:00.53 sh          \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m\u001b[1m  784 root      20   0   36856   3004   2592 R   0.0  0.1   0:00.00 top         \u001b[m\u001b[m\u001b[K\n",
      "\u001b[J\u001b[H\u001b[mtop - 12:25:42 up 4 days,  6:45,  0 users,  load average: 0.00, 0.02, 0.00\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "Tasks:\u001b[m\u001b[m\u001b[1m  10 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m   1 \u001b[m\u001b[mrunning,\u001b[m\u001b[m\u001b[1m   9 \u001b[m\u001b[msleeping,\u001b[m\u001b[m\u001b[1m   0 \u001b[m\u001b[mstopped,\u001b[m\u001b[m\u001b[1m   0 \u001b[m\u001b[mzombie\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "%Cpu(s):\u001b[m\u001b[m\u001b[1m  0.4 \u001b[m\u001b[mus,\u001b[m\u001b[m\u001b[1m  0.5 \u001b[m\u001b[msy,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mni,\u001b[m\u001b[m\u001b[1m 99.1 \u001b[m\u001b[mid,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mwa,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mhi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[msi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mst\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "KiB Mem :\u001b[m\u001b[m\u001b[1m  2046844 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m    82660 \u001b[m\u001b[mfree,\u001b[m\u001b[m\u001b[1m  1855168 \u001b[m\u001b[mused,\u001b[m\u001b[m\u001b[1m   109016 \u001b[m\u001b[mbuff/cache\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\u001b[K\n",
      "\n",
      "\u001b[m  759 root      20   0  562420  53140  13436 S   0.3  2.6   0:02.18 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m    1 root      20   0   18236    204    100 S   0.0  0.0   0:00.11 bash        \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  667 root      20   0  999296  56116   5060 S   0.0  2.7   0:21.62 jupyter-no+ \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  673 root      20   0  555440   8460   3128 S   0.0  0.4   0:24.88 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  689 root      20   0 2521820 1.492g  18052 S   0.0 76.4 760:54.45 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  711 root      20   0  555292  44516   6204 S   0.0  2.2   0:15.04 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  727 root      20   0  555292  44656   6280 S   0.0  2.2   0:15.52 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  743 root      20   0  555292  44644   6316 S   0.0  2.2   0:14.36 python      \u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\u001b[m\u001b[1m  784 root      20   0   36824   3028   2592 R   0.0  0.1   0:00.00 top         \u001b[m\u001b[m\u001b[K\n",
      "\u001b[J\u001b[H\u001b[mtop - 12:25:45 up 4 days,  6:45,  0 users,  load average: 0.00, 0.02, 0.00\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\n",
      "%Cpu(s):\u001b[m\u001b[m\u001b[1m  0.7 \u001b[m\u001b[mus,\u001b[m\u001b[m\u001b[1m  0.3 \u001b[m\u001b[msy,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mni,\u001b[m\u001b[m\u001b[1m 99.1 \u001b[m\u001b[mid,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mwa,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mhi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[msi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mst\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "KiB Mem :\u001b[m\u001b[m\u001b[1m  2046844 \u001b[m\u001b[mtotal,\u001b[m\u001b[m\u001b[1m    82660 \u001b[m\u001b[mfree,\u001b[m\u001b[m\u001b[1m  1855164 \u001b[m\u001b[mused,\u001b[m\u001b[m\u001b[1m   109020 \u001b[m\u001b[mbuff/cache\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\u001b[K\n",
      "\n",
      "\u001b[m  667 root      20   0  999296  56116   5060 S   0.3  2.7   0:21.63 jupyter-no+ \u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\u001b[m  673 root      20   0  555440   8460   3128 S   0.0  0.4   0:24.88 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  689 root      20   0 2521820 1.492g  18052 S   0.0 76.4 760:54.45 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  711 root      20   0  555292  44516   6204 S   0.0  2.2   0:15.04 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  727 root      20   0  555292  44656   6280 S   0.0  2.2   0:15.52 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  743 root      20   0  555292  44644   6316 S   0.0  2.2   0:14.36 python      \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  759 root      20   0  562420  53140  13436 S   0.0  2.6   0:02.18 python      \u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\n",
      "\u001b[J\u001b[H\u001b[mtop - 12:25:48 up 4 days,  6:45,  0 users,  load average: 0.00, 0.02, 0.00\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\n",
      "%Cpu(s):\u001b[m\u001b[m\u001b[1m  0.7 \u001b[m\u001b[mus,\u001b[m\u001b[m\u001b[1m  0.6 \u001b[m\u001b[msy,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mni,\u001b[m\u001b[m\u001b[1m 98.7 \u001b[m\u001b[mid,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mwa,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mhi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[msi,\u001b[m\u001b[m\u001b[1m  0.0 \u001b[m\u001b[mst\u001b[m\u001b[m\u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\n",
      "\u001b[K\n",
      "\n",
      "\u001b[m    1 root      20   0   18236    204    100 S   0.0  0.0   0:00.11 bash        \u001b[m\u001b[m\u001b[K\n",
      "\u001b[m  667 root      20   0  999296  56116   5060 S   0.0  2.7   0:21.63 jupyter-no+ \u001b[m\u001b[m\u001b[K\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\u001b[J\u001b[?1l\u001b>\u001b[25;1H\n",
      "\u001b[K"
     ]
    }
   ],
   "source": [
    "!top"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's look how class-balanced the data is... and given some data, \"carve\" a balanced training set out of it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import xgboost as xgb\n",
    "from xgboost import XGBClassifier\n",
    "import datetime; import pytz\n",
    "import matplotlib.pyplot as plt\n",
    "from scipy.special import softmax\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.model_selection import train_test_split # (*arrays, **options)\n",
    "import numpy as np\n",
    "from sklearn.metrics import log_loss\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
    "\n",
    "from joblib import dump, load\n",
    "import joblib\n",
    "import os\n",
    "from sklearn.metrics import confusion_matrix, mean_squared_error\n",
    "from sklearn.model_selection import KFold, train_test_split, GridSearchCV\n",
    "from sklearn.metrics import accuracy_score\n",
    "import fresh.utils as fu\n",
    "\n",
    "from importlib import reload\n",
    "from collections import Counter\n",
    "from tqdm.notebook import tqdm\n",
    "import fresh.preproc.v1 as pv1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\n",
    "model_loc = ('/opt/program/artifacts/2020-06-17T042025Z/model.xg')\n",
    "#model = joblib.load(model_loc)\n",
    "model = xgb.XGBClassifier()# .load_model(fname=model_loc)\n",
    "model.load_model(fname=model_loc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(X_test_transformed[:2].shape)\n",
    "# print(X_test_transformed[:2].toarray())\n",
    "# y_preds = model.predict_proba(X_test_transformed[:2])\n",
    "# print(y_preds.shape)\n",
    "# y_preds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quick test of parallel predict_proba "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "datadir = '/opt/data'\n",
    "localdir = '/opt/program'\n",
    "\n",
    "\n",
    "tripsdf = pd.read_csv(f'{datadir}/2013-07 - Citi Bike trip data.csv'\n",
    "                     ).sample(frac=0.017, random_state=42)\n",
    "stationsdf = pd.read_csv(f'{localdir}/datas/stations/stations-2018-12-04-c.csv',\n",
    "                        index_col=0)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X, y, neighborhoods = fu.prepare_data(tripsdf, stationsdf)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "# preproc\n",
    "(X_transformed,\n",
    "     one_hot_enc, le,\n",
    "     y_enc) = pv1.preprocess(X_train, y_train, # X[train_index]\n",
    "                         neighborhoods)\n",
    "labels = le.classes_\n",
    "\n",
    "# Test set\n",
    "X_test_transformed = one_hot_enc.transform(X_test)\n",
    "y_test_enc = le.transform(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_model = xgb.XGBClassifier(objective='multi:softprob'\n",
    "                             ).fit(X_transformed, \n",
    "                                        y_enc, verbose=True)\n",
    "\n",
    "# "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/opt/program/artifacts/2020-06-20T181230Z\n"
     ]
    }
   ],
   "source": [
    "workdir = fu.make_work_dir(); print(workdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3585, 78)"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_transformed.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/opt/program/artifacts/2020-06-20T181230Z/bundle.joblib']"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump({'model': xgb_model, 'note': 'just a quick test'}, f'{workdir}/bundle.joblib')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10755, 78)"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "#X_test_transformed.toarray().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6d7f99742a3c410796c2a7ec015b3175",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=9.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "results = []\n",
    "for i in tqdm(range(1, 10)):\n",
    "    X = np.concatenate([X_test_transformed.toarray()]*i)\n",
    "    start = datetime.datetime.now()\n",
    "    y_pred_1 = fu.predict_proba(X,\n",
    "                     model_loc='/opt/program/artifacts/2020-06-20T181230Z/bundle.joblib',\n",
    "                             parallel=False)\n",
    "    oneshot_time = fu.do_walltime(start)\n",
    "\n",
    "    start = datetime.datetime.now()\n",
    "    y_pred_2 = fu.predict_proba(X,\n",
    "                     model_loc='/opt/program/artifacts/2020-06-20T181230Z/bundle.joblib',\n",
    "                             parallel=True)\n",
    "    parallel_time = fu.do_walltime(start)\n",
    "    \n",
    "    results.append({'size': X.shape[0], 'parallel_time': parallel_time,\n",
    "                   'oneshot_time': oneshot_time})\n",
    "    # parallel=True\n",
    "    # 5.77s , 445ms\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.allclose(y_pred_1, y_pred_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ok cool... glad to see I have now added a new tool to my toolbox here, \n",
    "# For super small data, parallel overheads outweighs the benefit, but there is a tipping point..\n",
    "# my goal is for this to also help w/ not just filling and sitting on memory w/o releasing it! \n",
    "df = pd.DataFrame.from_records(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'seconds')"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3xV9f3H8dcnGxJIQggrIUM2hJUEUJElikAZlQpqcQ9arbZqHaitoK3WWvTn1qLWVUXrwMkQQcQBKEEUEJAZCAQSkpCQvb6/P84hRiQYQu49d3yej0ceuffccT4nF9755nu+5/sVYwxKKaX8R4DTBSillHIvDX6llPIzGvxKKeVnNPiVUsrPaPArpZSfCXK6gMZo27atSUpKcroMpZTyKhkZGQeNMbFHb/eK4E9KSmLNmjVOl6GUUl5FRDKPtV27epRSys9o8CullJ/R4FdKKT/jFX38x1JVVUVWVhbl5eVOl+LxwsLCiI+PJzg42OlSlFIewGuDPysri1atWpGUlISIOF2OxzLGkJeXR1ZWFsnJyU6Xo5TyAF7b1VNeXk5MTIyG/i8QEWJiYvQvI6VUHa8NfkBDv5H056SUqs+rg18ppXxWWQEsvA3KC5v9rTX4PdALL7zAddddB8Ds2bOZM2fOcZ/fmOcopbzIjk/hqaHw9bOQ+WWzv70Gv0Oqq6udLkEp5WmqymHRHfDSJAhuCVcugR7jmn03GvwnYdeuXfTs2ZPp06fTq1cvzjvvPEpLS7nnnnsYNGgQKSkpzJgxgyOrnI0cOZIbbriB9PR0HnnkEd5//32GDBnCwIEDOeusszhw4MBx97d9+3bGjh1LWloaw4YNY/Pmze44TKWUO+zfAM+MglVPwKCr4HcrIC7VJbvy2uGc9d39/ka+31fUrO/Zu1NrZk3s84vP27JlC8899xxDhw7liiuu4Mknn+S6667jrrvuAuDiiy/mgw8+YOLEiQBUVlbWzTtUUFDAqlWrEBGeffZZHnjgAR588MEG9zVjxgyefvppunXrxurVq7n22mtZtmxZMxytUsoxtbVW2C+9B8Ki4LdvQPcxLt2lTwS/kzp37szQoUMBuOiii3j00UdJTk7mgQceoLS0lPz8fPr06VMX/Oeff37da7Oysjj//PPJzs6msrLyuOPsi4uL+fLLL5k6dWrdtoqKChcdlVLKLQqzYP7vYddn0HMCTHwEwtu6fLc+EfyNaZm7ytFDJUWEa6+9ljVr1tC5c2dmz579kzH04eHhdbevv/56brrpJiZNmsTy5cuZPXt2g/upra0lKiqKdevWNfsxKKUcsP5N+PAmqKmGSY/BwIvBTUOvXdbHLyL/EZEcEdlQb9u/RGSziHwnIvNFJMpV+3eX3bt3s3LlSgBeffVVzjjjDADatm1LcXExb775ZoOvLSwsJC4uDoAXX3zxuPtp3bo1ycnJvPHGG4B1Re63337bHIeglHKnskPw5pXw1pXQtgdc8zmkXuK20AfXntx9ARh71LYlQIoxph/wA3C7C/fvFj169OCJJ56gV69eFBQUcM0113D11VeTkpLCOeecw6BBgxp87ezZs5k6dSppaWm0bfvLf9698sorPPfcc/Tv358+ffrw7rvvNuehKKVcbecKa5jmxvkw6k64fCG0OcXtZciREScueXORJOADY0zKMR47FzjPGDP9l94nPT3dHL0Qy6ZNm+jVq1czVdo0u3btYsKECWzYsOGXn+wwT/h5KeW3qitg2d/gy8etoJ/yDMSnuXy3IpJhjEk/eruTffxXAK839KCIzABmACQkJLirJqWUal4HNsLbM+DABki/Asb8HULCf/l1LuTIOH4RuROoBl5p6DnGmLnGmHRjTHps7M+WjPQISUlJXtHaV0o5oLYWVj4Bc0dB8QG48HWY8H+Ohz440OIXkcuACcBo48p+JqWUckrhXnjnGtj5KfQYDxMfhQjPacC6NfhFZCxwKzDCGFPqzn0rpZRbbHgLPrgRaqqscfmpl7p1xE5juCz4RWQeMBJoKyJZwCysUTyhwBJ7/PsqY8zvXVWDUkq5TXkhLLgFvnsd4tJhylyI6eJ0VcfksuA3xlx4jM3PuWp/SinlmF2fW1fgFu2DkbfDsJsh0HOvj9VJ2jxQ/WmZG+u+++5zUTVKqQZVV8CSu+CFCRAYDFd+BCNnenTogwa/z9DgV8rNcjbBM6Phi0cg7VL43WcQ/7Mh8x5Jg/8kPPTQQ6SkpJCSksLDDz/Mrl276NWrF1dffTV9+vRhzJgxlJWVAQ1PqfzGG2+QkpJC//79GT58eN1779u3j7Fjx9KtWzduvfXWuu3z5s2jb9++pKSkcNtttwEwc+ZMysrKGDBgANOn/+L1cEqpk1FbC6uegn+PgMPZcME86yRuaITTlTWaZ/890lgLZ8L+9c37nh36wrj7G3w4IyOD559/ntWrV2OMYciQIYwYMYKtW7cyb948nnnmGaZNm8Zbb73FRRdd1OCUyvfccw+LFy8mLi6OQ4cO1b3/unXr+OabbwgNDaVHjx5cf/31BAYGctttt5GRkUF0dDRjxozhnXfe4f777+fxxx/XCdyUcrWiffDOtbDjE+h2Dkx+HCLaOV3VCfON4HfA559/zrnnnls32+aUKVP47LPPSE5OZsCAAQCkpaWxa9eu406pPHToUC677DKmTZvGlClT6h4fPXo0kZGRAPTu3ZvMzEzy8vIYOXIkRy5omz59OitWrODXv/61W45ZKb+2cT68fwPUVFoXYqVd7nHDNBvLN4L/OC1zdwsNDa27HRgYSFlZ2XGnVH766adZvXo1H374IWlpaWRkZBzzfXSpRqUcUl5oLXr+7TzolGrNs9O2q9NVnRTt42+iYcOG8c4771BaWkpJSQnz589n2LBhx3zu8aZU3r59O0OGDOGee+4hNjaWPXv2NLjPwYMH8+mnn3Lw4EFqamqYN28eI0aMACA4OJiqqqpmPkql/Fzml/DUGdbY/BG3WaN2vDz0QYO/yVJTU7nssssYPHgwQ4YM4aqrriI6OrrB5zc0pfItt9xSd7L29NNPp3///g2+R8eOHbn//vsZNWoU/fv3Jy0tjcmTJwPWsoz9+vXTk7tKNYfqSvh4Njw/HgIC4IrFMOoOa8imD3DptMzNxVOnZfYm+vNSqpFyNsPbV8P+76xVscb+A0JbOV1Vk3jitMxKKeU5jIGv5loXZIWEw/mvQK8JTlflEhr8SilVlA3vXgvbl0HXs2HyE9CqvdNVuYxXB78x5meLnauf84buPKUcUVsD37xs9edXlcP4OTDoKq8dptlYXhv8YWFh5OXlERMTo+F/HMYY8vLyCAsLc7oUpTzLzs9g0e1wYD10PhUmPQax3Z2uyi28Nvjj4+PJysoiNzfX6VI8XlhYGPHx8U6XoZRnyN8JS/4Km96HyM5w3n+gzxSfb+XX57XBHxwcTHJystNlKKW8RcVh+OwhWPk4BATBqDvh9OshuIXTlbmd1wa/Uko1Sm2tddXt0ruttW/7XQBnzYLWnZyuzDEa/Eop37V7FSyaCfu+gfhBcMGrXjN1sitp8CulfM+hPfDxLGv921adrPl1Us6zrsJVGvxKKR9SWWItjPLFI9b94bfCGTdYF2SpOhr8SinvZwysfwOWzILD+6xROmffDVEJTlfmkTT4lVLeLSsDFt0GWV9DxwHW8MzE05yuyqNp8CulvFPRPvj4bvjuNYhoD5OfhP4Xaj9+I2jwK6W8S1UZfPk4fP6QNeXCGTfBsJu8dgZNJ2jwK6W8gzHW8odLZkHhbug1Ccb8DaKTnK7M62jwK6U837511rw6u7+E9n3h1x9A8rFXvFO/TINfKeW5Dh+AZffAN69AyxiY8DCkXgIBgU5X5tU0+JVSnqe6AlY9CSsehOpyOP06GH4LhEU6XZlPcFnwi8h/gAlAjjEmxd7WBngdSAJ2AdOMMQWuqkEp5WWMgc0fwEd/gYJd0GM8jPk7xHRxujKf4spxTy8AY4/aNhNYaozpBiy17yulFOzfAC9OhNcvgqAWcPF8uHCehr4LuKzFb4xZISJJR22eDIy0b78ILAduc1UNSikvUHIQPrkXMl6wunLGz4G0yyFQe6Jdxd0/2fbGmGz79n6gwUUtRWQGMAMgIUEvu1bK51RXwtfPwPJ/QmUxDJ4BI26Dlm2crsznOfYr1RhjRKTBxWCNMXOBuQDp6em6aKxSvsIY+GExfHQn5G2DrmfBOfdBbA+nK/Mb7g7+AyLS0RiTLSIdgRw3718p5aSczbD4Dti+FGK6wfQ3odvZTlfld9wd/O8BlwL329/fdfP+lVJOOLwflt8Pa1+C0AgYez8MugoCg52uzC+5cjjnPKwTuW1FJAuYhRX4/xORK4FMYJqr9q+U8gBlh6y58Vc9BbXVMOhKGDETwmOcrsyvuXJUz4UNPDTaVftUSnmIqnL4aq41kVpZAfSdai1u3ibZ6coUeuWuUqo51dZYC5t/8g8oyrJO3I6eBR37OV2ZqkeDXyl18oyBLQtg6T2Quxni0uDcp3UiNQ+lwa+UOjmZX8LHs2HPamukzrSXoddEEHG6MtUADX6lVNMc2GitgLV1MbTqBBMfhQHT9YpbL6CfkFLqxBRkwif3wXevQ1hrOOtuGPI7CG7hdGWqkTT4lVKNU3IQVsyBNc+BBMDQP8IZN0KLaKcrUydIg18pdXwVxbDyCfjyMagqgYEXWWPxI+Ocrkw1kQa/UurYqiutGTNXPAAludYJ2zPvgtjuTlemTpIGv1Lqp2prYePbsOxv1mIoScPgwtcgPt3pylQz0eBXSlmMsSZP+/hu2P+dtaj59Leg62gdmuljNPiVUpCVAR/Pgl2fQVQiTHkWUn4DAa5cpE85RYNfKX92cKt1te2m96BlWxj3gLX6VVCI05UpF9LgV8ofFe2zpkn+5r/W+PuRt8Npf4DQVk5XptxAg18pf1J2CL54GFY9bU2TPPhqGHYzRMQ6XZlyIw1+pfxBVZk1TfJnD0F5IfSbBqPugOgkpytTDtDgV8qX1VTDt69a3TpFe6Hr2XDWLOjQ1+nKlIM0+JXyRcbA5g9g6d/g4BaIS4cpcyHpDKcrUx5Ag18pX5O7Bd79A2R9DW27w/n/hZ4TdCy+qqPBr5Qv2foxvHk5BIXCpMeg/291mmT1M/ovQilfYAysfhoW3wHt+sBvX4PIeKerUh5Kg18pb1dTBQtutiZU6zkBzv03hEY4XZXyYBr8Snmz0nz43yXWVAvD/gyj/qLTLKhfpMGvlLfK/QHmnQ+FWVYrv/8FTlekvIQGv1LeaNtSeMOeU+fSDyBhiNMVKS+ifxMq5W1Wz4VXplonb69epqGvTpi2+JXyFjVVsPA2a83bHuOtC7J0UjXVBBr8SnmDsgL436Ww81MY+icYPQsCAp2uSnkpR4JfRG4ErgIMsB643BhT7kQtSnm8g9usk7gFmTD5SRg43emKlJdzex+/iMQBfwTSjTEpQCCgwxGUOpYdy+HZM60W/6Xva+irZuHUyd0goIWIBAEtgX0O1aGU5/r6OXh5CrTqZJ3ETTzN6YqUj2hU8ItIuIgE2Le7i8gkEQluyg6NMXuBOcBuIBsoNMZ8dIx9zhCRNSKyJjc3tym7Uso71VTDglvgw5ushc6v/EjnzVfNqrEt/hVAmN1N8xFwMfBCU3YoItHAZCAZ6ASEi8hFRz/PGDPXGJNujEmPjdXVgZSfKDsEr061Fk057Tq48DUIa+10VcrHNDb4xRhTCkwBnjTGTAX6NHGfZwE7jTG5xpgq4G3g9Ca+l1K+I287PHsW7Fxhzax5zr06cke5RGNH9YiInAZMB660tzX1X+Ru4FQRaQmUAaOBNU18L6V8w84V8PrFIAFwybu6YIpyqca2+G8AbgfmG2M2isgpwCdN2aExZjXwJrAWayhnADC3Ke+llE9Y8zy8fC5EtIerl2roK5cTY4zTNfyi9PR0s2aN/lGgfExNNXz0F1j9FHQZDVOfh7BIp6tSPkREMowx6UdvP25Xj4i8j3WR1TEZYyY1Q21K+Z/yQnjzSti2BIZcA2P+ritlKbf5pX9pc+zvU4AOwH/t+xcCB1xVlFI+LX8HvHoB5G+HCf8H6Vc4XZHyM8cNfmPMpwAi8uBRfy68LyLa96LUidr1Bbx+EZhauHg+JA93uiLlhxp7cjfcPqELgIgkA+GuKUkpH7X2JXhpMrSMsa7E1dBXDmlsp+KNwHIR2QEIkAj8zmVVKeVLamtgyV2w8nE4ZRRMfQFaRDldlfJjjQp+Y8wiEekG9LQ3bTbGVLiuLKV8RHkRvHUVbF0Mg2fAOf/Qk7jKcSfyLzANSLJf019EMMa85JKqlPIFBbusk7gHf4Dxc2Dw1U5XpBTQyOAXkZeBLsA6oMbebAANfqWOJXMlvD4daqvhoregyyinK1KqTmNb/OlAb+MNV3sp5bRvXoH3/wRRCfDb16FtN6crUuonGhv8G7DG8We7sBalvFttDSy9G754xBqxM/VFaNnG6aqU+pnGBn9b4HsR+QqoO6mrV+4qZas4DG/PgC0LrAuyxj0AgU1askIpl2ts8M92ZRFKebVDu2HehZDzPYz7l3USV8TpqpRqUGOHc34qIu2BQfamr4wxOa4rSykvsecreO23UF0B09+Armc5XZFSv6ixSy9OA74CpgLTgNUicp4rC1PKo9XWwup/wwu/gpAIuOpjDX3lNRrb1XMnMOhIK19EYoGPsebVV8q/5O+Ad6+HzM+h69kwZa6exFVepbHBH3BU104ejZ/nRynfUFsLXz8DH8+GgCCY/AQMmK79+crrNDb4F4nIYmCeff98YKFrSlLKA/2klX8WTHwUIuOcrkqpJmnsyd1bRGQKcGRNuLnGmPmuK0spD6GtfOWDGjtlQzKwwBjztn2/hYgkGWN2ubI4pRylrXzloxrb1fMGcHq9+zX2tkHHfrpSXkxb+crHNTb4g4wxlUfuGGMqRSTERTUp5Rxt5Ss/0NjgzxWRScaY9wBEZDJw0HVlKeVm2spXfqSxwf974BUReQJrOuYs4BKXVaWUO2krX/mZxo7q2Q6cKiIR9v1il1allDtoK1/5qcaO6mkP3Ad0MsaME5HewGnGmOdcWp1SrqKtfOXHGnv17QvAYqCTff8H4AZXFKSUSx2ZY+epobD/O6uVP/1NDX3lVxo9H78x5n8icjuAMaZaRGp+6UVKeRRt5SsFND74S0QkBuvELiJyKlDY1J2KSBTwLJBiv+cVxpiVTX0/pY7rJ335wdqXr/xeY4P/JuA9oIuIfAHEAiczLfMjwCJjzHn29QAtT+K9lGrY0TNpTnxEW/nK7zU2+LsA44DOwG+AISfw2p8QkUhgOHAZWBeDAZXHe41SJ0xb+Uo1qLEnd/9qjCkCooFRwJPAU03cZzKQCzwvIt+IyLMiEn70k0RkhoisEZE1ubm5TdyV8kv5O+DFibDwVkgcCteuhIEXaegrZWts8B85kfsr4BljzIdAU6dsCAJSgaeMMQOBEmDm0U8yxsw1xqQbY9JjY2ObuCvlV34yYme9PWLnDe3aUeooje2u2Ssi/wbOBv4pIqE0fSGWLCDLGLPavv8mxwh+pU6I9uUr1WiNDf5pwFhgjjHmkIh0BG5pyg6NMftFZI+I9DDGbAFGA9835b2U0r58pU5cY6dsKAXernc/G8g+if1ejzX3TwiwA7j8JN5L+Stt5SvVJE0amXOyjDHrgHQn9q18gLbylTopjgS/Uk2mrXylTpoGv/IO2spXqtlo8CvPl7cd3vujtvKVVzLGUFZVQ3FFNaUV1veSimpKKqsprqixbldUU1JRY2+r/sm228f3pF98VLPWpMGvPFdVOXz+f9ZXUJi28pVbGGMor6r9SUCX2AF9ZFtxRTWllUdvq6n3/B9fU1JZTa1p3L7DggOICA0iPDSI8JAgIkKDqG7si0+ABr/yTNuWwoKbrT79vlNhzL3Qqr3TVSkfU11Ty//WZPHyqkwOlVbWhXhjszY0yArqlqGBdUEd3TKEztEtCQ8NJDzU2tYyJIgI+/6Rbdb3QFqGHAn6QIICm3p51InR4FeepWgfLL4DNs6HmK5wybtwykinq1I+xhjD8i253LdgE1tziukXH8nQrm3tQP4xsMOPhHL9baFBRIRYYR/spqBubhr8yjPUVFsnb5fdC7VVMOovMPSPEBTqdGXKx2zcV8h9CzbxxbY8EmNa8tT0VMamdED8qAtRg185b8/X8OGN1vw6Xc+G8f+CNslOV6V8THZhGXMW/8Db32QR2SKYWRN7M31IIiFB3tlqPxka/Mo5pfmw9G7IeBFadYRpL0GvSXryVjWrw+VV/PvTHTzz2Q6MgRnDTuHaUV2JbBHsdGmO0eBX7mcMfDsPPvorlBXAaX+AkTMhtJXTlSkfUl1Ty7yv9/Dwkh/IK6lkUv9O3HJODzq30XWfNPiVe+Vshg9vgswvIH4wTPg/6JDidFXKhxhjWLoph38s3MT23BIGJ7Xhuct6MaBz846F92Ya/Mo9Kkvg0wdg5eNWy37iozDwYgjwv/5V5Trrswq5d8H3rNqRzyltw5l7cRpn927vVyduG0ODX7ne5gWw8DYo3A0DLoKz74HwGKerUj5k76Ey5izewvxv9tImPIR7JvfhwsEJXjvc0tU0+JXrHNptBf6WBdCuN1y+CBJPc7oq5UOKyqt48pPt/OeLnQBcM7IL14zsQusw/z1x2xga/Kr5VVfCqiesrh2wWvinXguB+p9RNY+qmlpeXb2bR5ZuJb+kkikD4/jzOT2Ii2rhdGleQYNfNa9dX1gnb3M3Q88JMPZ+iOrsdFXKRxhj+Oj7A9y/cDM7D5Zw6iltuHN8b/rGRzpdmlfR4FfNo+SgNTzz21chKgEufB16jHW6KuVD1u05xL0ffs/XuwroEhvOc5emc2bPdnritgk0+NXJqa2FtS9a8+RXlsCwP8OwmyFEx0qr5rEnv5QHFm/h/W/30TYihL//OoULBnV224RmvkiDXzVd9ndWt07W15A0DH71IMT2cLoq5SMKS6t4Yvk2XvhiFwEBcP2ZXfndiC5EhGpsnSz9CaoTV14En9wHX/0bWsbAuXOh3zSdakE1i8rqWv67KpNHl22lsKyK36TG8+cx3ekYqSdum4sGv2o8Y6zpkhffAYf3Q/oVMPqv0CLa6cqUDzDGsGjDfu5ftJnMvFKGdo3hjvG96NNJT9w2Nw1+1Th522HBLbB9KXTsD+e/AvFpTlelfMTa3QXc++EmMjIL6N4+gucvH8TI7rF64tZFNPjV8VWVwxcPw2cPQWAIjHsABl0FAYFOV6Z8QGZeCQ8s2sKH67OJbRXK/VP6cl5avJ64dTENftWw7cvgw5shfzuk/AbOuQ9adXC6KuUDDpVW8tiybby0chdBAQH8aXQ3Zgw/hXA9cesW+lNWP1eUbS9/+Da06QIXz4cuZzpdlfIBFdU1vLwyk0eXbuVwRTXT0jpz05jutG8d5nRpfkWDX/2otga+egaW/R1qKmHkHTD0TxCs/ylV0xhj2FdYztrMAtbuLmDJ9wfIKihjePdYbh/Xk14dWztdol/S4FeWg9vgnd9bY/K7jLaWP4zp4nRVystUVNewYW8R3+wuIMMO+wNFFQCEBQcwsHM0953bl+HdYx2u1L85FvwiEgisAfYaYyY4VYffq621FjlfMsta2Pw3z1n9+TqaQjVCdmEZazMPsXa3FfIb9xZRWVMLQHx0C4Ykx5CaEEVqYjS9OrbWaZI9hJMt/j8BmwD9W88ph/bAu9fCzhXWIueTHoPWHZ2uSnmoiuoaNu4rYm1mAd/stsI+u7AcgNCgAPrFR3L50CQGJkSTmhBFO+2391iOBL+IxAO/Au4FbnKiBr9mDKx7FRbNBFNrrYaVeom28tVP7C8st1rydpfNhn1FVFZbrfm4qBakJUaTmhBNmt2aDwnS1ry3cKrF/zBwK9Dg6toiMgOYAZCQkOCmsvzA4QPwwQ3W4iiJQ+HXT0J0ktNVKYdVVtfyfXZRXcivzSxgn92aDwkKoG9cJJeelkhqQjSpidE6CsfLuT34RWQCkGOMyRCRkQ09zxgzF5gLkJ6ebtxUnm/b+A58cKM1i+Y598GQa3TNWz+VU2S35ncfYm1mAev3FlJht+Y7RYYxMDGaK+0um96dWhMapBfs+RInWvxDgUkiMh4IA1qLyH+NMRc5UIt/KM2HhbfC+jeg00A49986i6Yfqaqp5ft9RT8J+r2HygAICQwgJa41F5+aSKrdddMhUlvzvs7twW+MuR24HcBu8d+soe9CW5fAe9dDSS6MuhPOuFGXQPRxxRXVfLntIBl2l813WT+25ju0DiM1MaruJGxKnLbm/ZGO4/dVFYfho79AxgsQ2wsufA06DXC6KuUiReVVLN10gAXr9/PpD7lUVtcSHCj06RTJ9CGJpCZGkZoQTSddk1bhcPAbY5YDy52swSft+gLeuQYO7YbT/2i19PXqW59zqLSSJd8fYOGG/Xy2NZeqGkOH1mFMH5LAOX06MKBzFGHB2ppXP6ctfl9SVQ7L/gYrn4DoRLh8ISSe5nRVqhnlFVfwkR32X247SHWtIS6qBZednsS4vh0ZEB9FQIAOy1XHp8HvK/auhfm/h4NbrAVSzv4bhEY4XZVqBjmHy1m88QAL12ezakcetQYS2rTkqmGnML5vB/rGReq89eqEaPB7u5oqWDEHVvwLItrDRW9B17OcrkqdpP2F5SzakM2CDfv5elc+xsApbcO5dmRXxvXtQO+OrTXsVZNp8HuznM0w/3eQvQ76nQ/j/qnLIHqxvYfKWLg+m4Ub9pORWQBA9/YR/PHMbozv25Hu7SM07FWz0OD3RrU1Vj/+sr9b3TnTXobek5yuSjXB7rxSFtot+2/3HAKgV8fW/Pns7ozr24Gu7Rq8uF2pJtPg9zb5O+Gda2H3l9DjVzDxYYho53RV6gTsPFjCgvXZLNyQzYa9RQD0jYvk1rE9GJfSkeS24Q5XqHydBr+3MAYynofFf7HWu/31U9D/Qp1YzUtsyznMgvX7WbA+m837DwMwoHMUd4zvybiUjnRu09LhCpU/0eD3BkX74N3rYPtSSB4Bk5+AqM5OV6WOwxjDlgNW2C9cn83WnGJEIC0hmr9O6M3YlPXfvK0AAA7SSURBVA7E6cVUyiEa/J7MGGt+nQU3Q3UljJ8D6VfqxGoeyhjDxn1FLNyQzcL1+9lxsAQRGJzUhnsm9+GcPh10VkvlETT4PVXJQWsmzU3vQfxgOPdpXQrRAxlj+C6rkAV22O/OLyUwQDjtlBiuHJbMmN4diG0V6nSZSv2EBr8n2rwA3v8jlBfCWbOtaRcC9NJ7T1Bba9iWW2ytJ5tZwJfb89h7qIygAGFo17b8YVQXzu7dgTbhIU6XqlSDNPg9SXkhLLod1r0C7fvCJe9C+z5OV+XXiiuqWWcvM3hk8fDD5dUARLcMJj2pDTee3Z2ze7UnsqXOeqq8gwa/p9ixHN75AxzeB8NuhhG3QZC2Gt3JGMPu/NK6gM/IPMSW/UXUGmvwVPd2rZjQrxOpCVGkJUaT3DZcL6hSXkmD32mVpfDxLPhqLsR0hSuXQHy601X5hfKqGtbvLazrtlm7u4CDxZUARIQGMTAhijFndiM1MZoBnaOIbKEteuUbNPidtOcra2K1/O0w5PcwehaE6HhuV9lfWF6vNV/Axn2FVNVYq3omxbRkePfYugXEu7dvRaDOcql8lAa/E6orYPn98MXD0DoOLn0fkoc7XZVPqaqpZVN2kR30P11uMDQogP7xUVx5ximkJkSRmhhN2wgdeaP8hwa/u21dAotmQt42GHgRnPMPCGvtdFVeL7+ksq67JiOzgG+zDlFeZS032DEyjNTEaK48I5nUxGh6d2xNSJBeC6H8lwa/u+Rth8V3wA+LoE0XmP4WdNPpk5uittawNae4rttmbWYBOw6WABAUIPTp1JoLByfUddvocoNK/ZQGv6tVFMNnD8LKxyEwBM66G069VkfsnICyyhrWZObXddt8U29IZZvwEFITopma3pm0xGj6xkXSIkSveVDqeDT4XcUY2PAWfPRXa4hmvwusi7Fad3S6Mq+wJ7+UT7bksGxzDiu351FRXYsI9Gjfion9O5GWEE1aYjSJMS11SKVSJ0iD3xWyv4OFt8LuldCxP0x9ARKGOF2VR6uqqWXNrgKW22G/NacYsEbb/HZIAiO6x5KaGE3rMB1SqdTJ0uBvTqX51uIoGc9bK2FNfAQGXqzTLTTgYHEFy7fk8snmHFZszeVweTXBgcLg5DacP6gzZ/Zsxymxum6wUs1Ng7851FRbYb/s71BxGAbPgJEzdRnEo9TWWrNXLtucw7ItOXyXdQhjoF2rUMandGRUz1jO6BZLRKj+s1TKlfR/2Mna9TksvA0ObICkYda6tzq/Tp3iimo+35rLss05fLIll9zDFYhA//gobjyrO2f2bEfvjq0J0IullHIbDf6mKsyCJXdZJ3AjO8PUF6H3ZL9fEcsYw46DJXyyOYdPtuTw1c58qmoMrcKCGN49ljN7tGNEj1i9YEopB2nwn6iqclj5GHz2EJhaGDEThv7Jr6daqKiuYfWOfLtVn0NmXikA3dtHcMUZyYzq0Y60xGiCA/WiKaU8gQZ/YxkDWxbC4tuhYBf0mghj7oXoRKcrc8T+wvK64ZZfbDtIaWUNoUEBnN4lhqvOSGZkj3a6jqxSHsrtwS8inYGXgPaAAeYaYx5xdx0nJPcHa5qF7UuhbQ+4+B3oMsrpqtyqptawbk+BdWJ2cy6bsosAiItqwZTUOM7s2Y7TTmmrF08p5QWcaPFXA382xqwVkVZAhogsMcZ870Atx1deBCsegFVPQXBLa16dwVdDoH+MJT9UWsmnP1jDLT/9IZeC0ioCA4S0xGhmjuvJqB7t6N4+Qi+gUsrLuD34jTHZQLZ9+7CIbALiAM8J/tpa+O41WDILSnKtydRGz4KIWKcrc6mqmlq25xZbffWbc8jILKDWWNMijOrRjlE92zG8W6yuNKWUl3O0j19EkoCBwOpjPDYDmAGQkJDgvqL2ZsCCW2HvGohLh9++BnFp7tt/MyqvqiG/pJL8kkrySirJL6kgr7jyqG327eIKiuz5bwD6dGrNH0Z1ZVTPdvSPj9K56ZXyIY4Fv4hEAG8BNxhjio5+3BgzF5gLkJ6ebpqyj4eW/MD73+4jPDSQ8JAgIkKDCLe/IkID7e9BtAwJIppD9Nv8CJ12vkV1i7YcGv0wpv/5RISF0MIYj+jOKK2sJq/4+CF+5LH84kpKKmuO+T5BAUJ0eAgx4SG0CQ+hT6fW9u1QOkWFMbx7LO1bh7n56JRS7uJI8ItIMFbov2KMedtV+0lo05KUuEhKKqoprqhmf1G5fbuGkopqyqpqCKKaSwKXcEPQW7Sggrk143ms4FyKP2wJH35i1wvhIUHWLxD7l4V13/oF0vIn2wLrfsH8+IsmsN7zgwgLtoY1Hq6oJr+4fsu7wrpdfIwWeUlF3fzyRwsJDKCNHeIxESEkxbS0btthHhPxY8jHhIfSukWQR/wiU0o5Q4xpUmO66Tu0EudFIN8Yc0NjXpOenm7WrFnT7LXUbFsOi24l8OAWSjuPYPfgu8hvkURxRTUllT/+giit98uiuLKakorqn/wCKbGf31AwHy0wQAgQ6pb9O1qL4MC6EK8L9CMhbt9vUy/MI0I1yJVSPyciGcaYny3i7USLfyhwMbBeRNbZ2+4wxixwWwUFmfDRnQRueh+ik+CCebTsMY6eJxme1TW1lFTW1PvFUE1JRY39vZrSer9MaoyhTcufh3hMeKgOiVRKuZQTo3o+B5xpnlaWwhePWGvdSgCc+Rc47XoIbp7+7KDAACJbBBDZQke9KKU8l39cuWsMfP8ufPQXKNwDKb+Bs++ByHinK1NKKbfz/eA/8D0sug12roD2KXDuvyFpqNNVKaWUY3w7+D/9Fyz/B4S2gvFzIO1yCPTtQ1ZKqV/i2ykYnQRpl8KZf4WWbZyuRimlPIJvB3+/qdaXUkqpOjpBulJK+RkNfqWU8jMa/Eop5Wc0+JVSys9o8CullJ/R4FdKKT+jwa+UUn5Gg18ppfyM2+fjbwoRyQUyHdh1W+CgA/t1JT0m76DH5B08/ZgSjTE/WyzcK4LfKSKy5liLGHgzPSbvoMfkHbz1mLSrRyml/IwGv1JK+RkN/uOb63QBLqDH5B30mLyDVx6T9vErpZSf0Ra/Ukr5GQ1+pZTyM34X/CKyS0TWi8g6EVljb2sjIktEZKv9PdreLiLyqIhsE5HvRCS13vtcaj9/q4hc6sBx/EdEckRkQ71tzXYcIpJm/5y22a8Vh45ptojstT+vdSIyvt5jt9v1bRGRc+ptH2tv2yYiM+ttTxaR1fb210UkxMXH01lEPhGR70Vko4j8yd7utZ/TcY7Jaz8ne59hIvKViHxrH9fdx6tFRELt+9vsx5OaeryOMMb41RewC2h71LYHgJn27ZnAP+3b44GFgACnAqvt7W2AHfb3aPt2tJuPYziQCmxwxXEAX9nPFfu14xw6ptnAzcd4bm/gWyAUSAa2A4H213bgFCDEfk5v+zX/Ay6wbz8NXOPi4+kIpNq3WwE/2HV77ed0nGPy2s/J3o8AEfbtYGC1/XM9Zi3AtcDT9u0LgNeberxOfPldi78Bk4EX7dsvAr+ut/0lY1kFRIlIR+AcYIkxJt8YUwAsAca6s2BjzAog/6jNzXIc9mOtjTGrjPWv+aV67+UyDRxTQyYDrxljKowxO4FtwGD7a5sxZocxphJ4DZhst4TPBN60X1//5+MSxphsY8xa+/ZhYBMQhxd/Tsc5poZ4/OcEYP/Mi+27wfaXOU4t9T/DN4HRdu0ndLwuPqwG+WPwG+AjEckQkRn2tvbGmGz79n6gvX07DthT77VZ9raGtjutuY4jzr599HanXGd3ffznSLcIJ35MMcAhY0z1Udvdwu4KGIjVkvSJz+moYwIv/5xEJFBE1gE5WL9ctx+nlrr67ccL7dq9IjP8MfjPMMakAuOAP4jI8PoP2i0nrx/j6ivHATwFdAEGANnAg86Wc+JEJAJ4C7jBGFNU/zFv/ZyOcUxe/zkZY2qMMQOAeKwWek+HS3IZvwt+Y8xe+3sOMB/rAz5g/9mM/T3HfvpeoHO9l8fb2xra7rTmOo699u2jt7udMeaA/R+yFngG6/OCEz+mPKyuk6CjtruUiARjBeQrxpi37c1e/Tkd65i8/XOqzxhzCPgEOO04tdTVbz8eiVW7V2SGXwW/iISLSKsjt4ExwAbgPeDISIlLgXft2+8Bl9ijLU4FCu0/0RcDY0Qk2v6Tdoy9zWnNchz2Y0Uicqrdb3lJvfdyqyMBaTsX6/MC65gusEdXJAPdsE50fg10s0djhGCdeHvPbll/Apxnv77+z8dVtQvwHLDJGPNQvYe89nNq6Ji8+XOy648VkSj7dgvgbKzzFw3VUv8zPA9YZtd+Qsfr6uNqkFNnlZ34wjqj/q39tRG4094eAywFtgIfA23Mj2f6n8Dq61sPpNd7ryuwTtxsAy534FjmYf1JXYXVX3hlcx4HkI71n3c78Dj2Vd4OHNPLds3fYf1H6Vjv+Xfa9W2h3mgWrNExP9iP3XnU5/+VfaxvAKEuPp4zsLpxvgPW2V/jvflzOs4xee3nZO+zH/CNXf8G4K7j1QKE2fe32Y+f0tTjdeJLp2xQSik/41ddPUoppTT4lVLK72jwK6WUn9HgV0opP6PBr5RSfkaDX6kTICLPikhvp+tQ6mTocE6llPIz2uJXqgH2ld4f2nO0bxCR80VkuYiki8gk+XHu+S0istN+TZqIfGpPArj4qCtalfIIGvxKNWwssM8Y098YkwIsOvKAMeY9Y8wAY03q9S0wx57D5jHgPGNMGvAf4F4nClfqeIJ++SlK+a31wIMi8k/gA2PMZ3LUAlcicitQZox5QkRSgBRgif28QKwpKJTyKBr8SjXAGPODWMsfjgf+LiJL6z8uImcBU7FWDgNrrp2NxpjT3FupUidGu3qUaoCIdAJKjTH/Bf6FtSzkkccSsSZUm2qMKbM3bwFiReQ0+znBItLHzWUr9Yu0xa9Uw/oC/xKRWqwZQ68B5tiPXYY1y+Y7drfOPmPMeBE5D3hURCKx/n89jDUTrFIeQ4dzKqWUn9GuHqWU8jMa/Eop5Wc0+JVSys9o8CullJ/R4FdKKT+jwa+UUn5Gg18ppfzM/wPYQx6rKI7u8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df['size'].tolist(), df.parallel_time.tolist(), label='parallel')\n",
    "plt.plot(df['size'].tolist(), df.oneshot_time.tolist(), label='oneshot')\n",
    "plt.legend()\n",
    "plt.xlabel('size')\n",
    "plt.ylabel('seconds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<module 'fresh.utils' from '/opt/program/fresh/utils.py'>"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reload(fu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "slices = fu.get_slices(range(200), slice_size=100)\n",
    "vec = [xgb_model.predict_proba(X_test_transformed[a:b]) for (a,b) in slices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0, 443], [444, 887], [888, 999]]\n"
     ]
    }
   ],
   "source": [
    "reload(fu)\n",
    "print(fu.get_slices(range(1000), slice_size=444))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10 ms, sys: 0 ns, total: 10 ms\n",
      "Wall time: 10.1 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0, 4, 12, 24, 40, 60, 84, 112, 144, 180]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#help(joblib)\n",
    "import time\n",
    "from joblib import Parallel, delayed\n",
    "from math import sqrt\n",
    "\n",
    "def mywork(x, y, z=None, a=None):\n",
    "    time.sleep(5)\n",
    "    \n",
    "    return x*y * (z or a)\n",
    "\n",
    "%time Parallel(n_jobs=5)(delayed(mywork)(i, i+1, z=2) for i in range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 660 ms, sys: 50 ms, total: 710 ms\n",
      "Wall time: 725 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "y_prob_vec = np.random.random(size=(y_test_enc.shape[0],\n",
    "                       labels.shape[0]))\n",
    "loss = fu.big_logloss(y_test_enc, y_prob_vec, list(range(len(labels))), parallel=False)\n",
    "# Wall time: 56.5 s\n",
    "# Wall time: 906 ms  . huh?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss#labels#y_test_enc.shape, y_prob_vec.shape#loss # 4.291267319642341\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((210854,), (54,))"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test_enc.shape, labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.21080369, 0.43961201, 0.19590716, ..., 0.71289874, 0.33170714,\n",
       "        0.82601838],\n",
       "       [0.36605098, 0.13483735, 0.90106828, ..., 0.54691864, 0.4643664 ,\n",
       "        0.88128189],\n",
       "       [0.63243054, 0.16359919, 0.66125254, ..., 0.34549834, 0.41448938,\n",
       "        0.6333276 ],\n",
       "       ...,\n",
       "       [0.14012141, 0.33125609, 0.49782631, ..., 0.26443373, 0.86010273,\n",
       "        0.96617081],\n",
       "       [0.50237769, 0.86188519, 0.61262241, ..., 0.67107822, 0.36280706,\n",
       "        0.08816574],\n",
       "       [0.02359937, 0.74809099, 0.15765813, ..., 0.00790811, 0.13526414,\n",
       "        0.87367221]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.random(size=(y_test_enc.shape[0],\n",
    "                       labels.shape[0]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}