mnist_evo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "USSV_OlCFKOD",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Pruning a neural network by evolutionary algorithm\n",
    "\n",
    "Author: Richard Klem\n",
    "\n",
    "E-mail: xklemr00@fit.vutbr.cz\n",
    "\n",
    "Date: 8.5.2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "J8y9ZkLXmAZc",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Copyright 2020 The TensorFlow Datasets Authors, Licensed under the Apache License, Version 2.0\n",
    "This copyright applies on the sections specifically annotated by _TAKEN FROM TF DEMO_.\n",
    "Other code is my (see author at the top of the source code) own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Libraries\n",
    "Import required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T08:54:18.253377Z",
     "iopub.status.busy": "2022-02-10T08:54:18.252747Z",
     "iopub.status.idle": "2022-02-10T08:54:20.347380Z",
     "shell.execute_reply": "2022-02-10T08:54:20.346741Z"
    },
    "id": "TTBSvHcSLBzc",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "from itertools import permutations\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "from adjustText import adjust_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VjI6VgOBf0v0",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Create your input pipeline\n",
    "\n",
    "Start by building an efficient input pipeline using advices from:\n",
    "* The [Performance tips](https://www.tensorflow.org/datasets/performances) guide\n",
    "* The [Better performance with the `tf.data` API](https://www.tensorflow.org/guide/data_performance#optimize_performance) guide\n",
    "\n",
    "_TAKEN FROM TF DEMO_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3aH3vP_XLI8",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Load a dataset\n",
    "\n",
    "Load the MNIST dataset with the following arguments:\n",
    "\n",
    "* `shuffle_files=True`: The MNIST data is only stored in a single file, but for larger datasets with multiple files on disk, it's good practice to shuffle them when training.\n",
    "* `as_supervised=True`: Returns a tuple `(img, label)` instead of a dictionary `{'image': img, 'label': label}`.\n",
    "\n",
    "_TAKEN FROM TF DEMO_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T08:54:20.352400Z",
     "iopub.status.busy": "2022-02-10T08:54:20.351836Z",
     "iopub.status.idle": "2022-02-10T08:54:21.267974Z",
     "shell.execute_reply": "2022-02-10T08:54:21.268411Z"
    },
    "id": "ZUMhCXhFXdHQ",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# TAKEN FROM TF DEMO\n",
    "(ds_train, ds_test), ds_info = tfds.load(\n",
    "    'mnist',\n",
    "    split=['train', 'test'],\n",
    "    shuffle_files=True,\n",
    "    as_supervised=True,\n",
    "    with_info=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rgwCFAcWXQTx",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Build a training pipeline\n",
    "\n",
    "Apply the following transformations:\n",
    "\n",
    "* `tf.data.Dataset.map`: TFDS provide images of type `tf.uint8`, while the model expects `tf.float32`. Therefore, you need to normalize images.\n",
    "* `tf.data.Dataset.cache` As you fit the dataset in memory, cache it before shuffling for a better performance.<br/>\n",
    "__Note:__ Random transformations should be applied after caching.\n",
    "* `tf.data.Dataset.shuffle`: For true randomness, set the shuffle buffer to the full dataset size.<br/>\n",
    "__Note:__ For large datasets that can't fit in memory, use `buffer_size=1000` if your system allows it.\n",
    "* `tf.data.Dataset.batch`: Batch elements of the dataset after shuffling to get unique batches at each epoch.\n",
    "* `tf.data.Dataset.prefetch`: It is good practice to end the pipeline by prefetching [for performance](https://www.tensorflow.org/guide/data_performance#prefetching).\n",
    "\n",
    "_TAKEN FROM TF DEMO_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T08:54:21.296628Z",
     "iopub.status.busy": "2022-02-10T08:54:21.274848Z",
     "iopub.status.idle": "2022-02-10T08:54:21.316234Z",
     "shell.execute_reply": "2022-02-10T08:54:21.315707Z"
    },
    "id": "haykx2K9XgiI",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# _TAKEN FROM TF DEMO_\n",
    "def normalize_img(image, label):\n",
    "    \"\"\"Normalizes images: `uint8` -> `float32`.\"\"\"\n",
    "    return tf.cast(image, tf.float32) / 255., label\n",
    "\n",
    "\n",
    "ds_train = ds_train.map(\n",
    "    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "ds_train = ds_train.cache()\n",
    "ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)\n",
    "ds_train = ds_train.batch(128)\n",
    "ds_train = ds_train.prefetch(tf.data.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RbsMy4X1XVFv",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Build an evaluation pipeline\n",
    "\n",
    "Your testing pipeline is similar to the training pipeline with small differences:\n",
    "\n",
    " * You don't need to call `tf.data.Dataset.shuffle`.\n",
    " * Caching is done after batching because batches can be the same between epochs.\n",
    "\n",
    "_TAKEN FROM TF DEMO_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-02-10T08:54:21.321408Z",
     "iopub.status.busy": "2022-02-10T08:54:21.320618Z",
     "iopub.status.idle": "2022-02-10T08:54:21.329812Z",
     "shell.execute_reply": "2022-02-10T08:54:21.330226Z"
    },
    "id": "A0KjuDf7XiqY",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# TAKEN FROM TF DEMO\n",
    "ds_test = ds_test.map(\n",
    "    normalize_img, num_parallel_calls=tf.data.AUTOTUNE)\n",
    "ds_test = ds_test.batch(128)\n",
    "ds_test = ds_test.cache()\n",
    "ds_test = ds_test.prefetch(tf.data.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nTFoji3INMEM",
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Create and train the model\n",
    "Swish function is smarter ReLU: https://towardsdatascience.com/swish-booting-relu-from-the-activation-function-throne-78f87e5ab6eb\n",
    "\n",
    "We can use SparseCategoricalAccuracy as we have non-overlapping categories. It will save some computational time and memory.\n",
    "\n",
    "See architecture bellow.\n",
    "![SNOWFALL](matlab_schema.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_model():\n",
    "    model = tf.keras.models.Sequential([\n",
    "        tf.keras.layers.Conv2D(filters=3, kernel_size=(3, 3), activation='swish', input_shape=(28, 28, 1)),\n",
    "        tf.keras.layers.AveragePooling2D(),\n",
    "        tf.keras.layers.Conv2D(filters=6, kernel_size=(3, 3), activation='swish'),\n",
    "        tf.keras.layers.AveragePooling2D(),\n",
    "        tf.keras.layers.Flatten(),\n",
    "        tf.keras.layers.Dense(10)\n",
    "    ])\n",
    "    model.compile(\n",
    "        optimizer=tf.keras.optimizers.Adam(0.005),\n",
    "        loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "        metrics=[tf.keras.metrics.SparseCategoricalAccuracy()],\n",
    "    )\n",
    "\n",
    "    model.fit(\n",
    "        ds_train,\n",
    "        epochs=5,\n",
    "        validation_data=ds_test\n",
    "    )\n",
    "    return model\n",
    "\n",
    "model = get_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Model architecture\n",
    "We can inspect the model architecture in code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "There is analyzed structure from Matlab's Deep Network Analyzer.\n",
    "You can see we have $150*10=1500$ trainable parameters in the dense layer.\n",
    "Hence the rest of the model has about 200 parameters, we will aim to reduce insignificant or less significant weights in the dense layer by evolutionary algorithm.\n",
    "![SNOWFALL](matlab_analyze.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Individual class and some methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Individual class\n",
    "Create a class to represent one individual \"solution\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class Individual:\n",
    "    def __init__(self, threshold):\n",
    "        super(Individual, self).__init__()\n",
    "        self.threshold = threshold\n",
    "        self.acc = 0\n",
    "        self.w_size = np.inf\n",
    "        self.fitness = 0\n",
    "\n",
    "    def __repr__(self):\n",
    "        return self.__str__()\n",
    "\n",
    "    def __str__(self):\n",
    "        return f\"threshold={round(self.threshold, 4)}, acc={round(self.acc, 4)}, w_size={self.w_size}, fitness={round(self.fitness, 4)}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Fitness\n",
    "Function to evaluate fitness of the individual based on accuracy and model size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_fitness(size, acc, baseline_w_size, baseline_acc, ):\n",
    "    return (acc / baseline_acc) ** 2 + (1 - size / baseline_w_size) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Mating -- cross-mutation\n",
    "Function to get child from parents – genetic cross-over."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def mating(mother: Individual, father: Individual):\n",
    "    return Individual((mother.threshold + father.threshold) / 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Eval model size and accuracy\n",
    "Function to get size and accuracy of the certain model. `w_size` like _weights_size_ as we are optimizing only weights, not biases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def eval_phenotype(model):\n",
    "    res = model.evaluate(ds_test, return_dict=True)\n",
    "    w_size = np.count_nonzero(model.layers[5].weights[0])\n",
    "    acc = res.get('sparse_categorical_accuracy')\n",
    "    return w_size, acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Population\n",
    "Function which creates one population based on `population_size`, `min_threshold` and `max_threshold` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def get_population(population_size=20, min_threshold=0, max_threshold=np.finfo(np.float64).max):\n",
    "    # We want to start with rather lower thresholds, so thus the logspace.\n",
    "    population = np.logspace(min_threshold, max_threshold, population_size - 1, base=16)\n",
    "    # Normalize the thresholds.\n",
    "    population = [min_threshold] + list(population / max(population) * max_threshold)\n",
    "    # Create Individual instances.\n",
    "    population = [Individual(threshold) for threshold in population]\n",
    "    return population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Epoch\n",
    "Make a function to simulate one epoch of the EA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def one_epoch(population: [Individual], min_t, max_t, base_w_size, base_acc):\n",
    "    population_size = len(population)\n",
    "    num_of_children = int(population_size // 4)\n",
    "    num_of_randoms = int(population_size // 20)\n",
    "    num_of_elites = population_size - num_of_children - num_of_randoms\n",
    "    mating_possibilities = list(permutations(population, 2))\n",
    "    pairs = random.sample(mating_possibilities, num_of_children)\n",
    "    children: [Individual] = [mating(x, y) for (x, y) in pairs]\n",
    "\n",
    "    pop = dict(zip(range(0, population_size), population))\n",
    "\n",
    "    for one in pop.values():\n",
    "        weights = orig_weights.copy()\n",
    "        # Calculate new weights according to the threshold value.\n",
    "        weights = weights * (abs(weights) > one.threshold)\n",
    "        # Update the model.\n",
    "        model.layers[5].set_weights([weights, orig_biases])\n",
    "        # Evaluate the new model with the new weights.\n",
    "        one.w_size, one.acc = eval_phenotype(model)\n",
    "        one.fitness = get_fitness(one.w_size, one.acc, base_w_size, base_acc)\n",
    "\n",
    "    # Filter elite individuals - the best ones.\n",
    "    elites: [Individual] = list(dict(sorted(pop.items(), key=lambda x: x[1].fitness, reverse=True)[:num_of_elites]).values())\n",
    "\n",
    "    # Get some randomly set individual(s) to discover something new.\n",
    "    randoms: [Individual] = []\n",
    "    for one in random.sample(list(pop.values()), num_of_randoms):\n",
    "        print(one.threshold)\n",
    "        one.threshold = np.random.uniform(min_t, max_t)\n",
    "        randoms.append(one)\n",
    "        print(one.threshold)\n",
    "\n",
    "    new_population = elites + children + randoms\n",
    "    return new_population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Evolution\n",
    "Function that runs N epochs of the EA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def run(population, min_t, max_t, base_w_size, base_acc, epochs=30):\n",
    "    for i in range(epochs):\n",
    "        population = one_epoch(population, min_t, max_t, base_w_size, base_acc)\n",
    "    return population"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Init a population and required variables\n",
    "Create an initial population and set required variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "base_w_size, base_acc = eval_phenotype(model)\n",
    "orig_weights = np.array(model.layers[5].weights[0])\n",
    "orig_biases = np.array(model.layers[5].weights[1])\n",
    "weights = orig_weights.copy()\n",
    "model.layers[5].set_weights([weights, orig_biases])\n",
    "\n",
    "population_size = 20\n",
    "min_threshold = 1e-6\n",
    "max_threshold = max(abs(np.amin(orig_weights)), abs(np.amax(orig_weights)))\n",
    "max_size = weights.shape[0] * weights.shape[1]\n",
    "\n",
    "population = get_population(population_size, min_threshold, max_threshold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Run the evolution\n",
    "Let's run 30 epochs of the EA. Each time, the population is updated according to the previously defined methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "epochs = 30\n",
    "population = run(population, min_threshold, max_threshold, base_w_size, base_acc, epochs=epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Pareto front\n",
    "Create data for plotting a pareto front.\n",
    "We plot top 75% solutions, the rest 25% are new children, so we cannot plot them as they were not evaluated.\n",
    "We want to see points on the top, on the right and on the frontier which will be facing towards top-right corner."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "output = list(sorted(population, key=lambda x: x.fitness, reverse=True)[:int(population_size * 0.75)])\n",
    "labels = ['acc', 'w_size', 'fitness', 'threshold']\n",
    "data = []\n",
    "\n",
    "for ind in output:\n",
    "    if ind.w_size > 10 and ind.acc > 0.5:\n",
    "        data.append([ind.acc, max_size/ind.w_size, ind.fitness, ind.threshold])\n",
    "\n",
    "data = pd.DataFrame(data, columns=labels)\n",
    "plt.title(\"Pareto front\")\n",
    "plt.xlabel(\"Accuracy\")\n",
    "plt.ylabel(\"Compression\")\n",
    "\n",
    "x_labels = []\n",
    "for i in range(len(data['threshold'])):\n",
    "    x_labels.append(plt.text(data['acc'][i], data['w_size'][i], str(round(data['threshold'][i], 2))))\n",
    "\n",
    "sns.scatterplot(data=data, x='acc', y='w_size')\n",
    "adjust_text(x_labels, x=list(data['acc']), y=list(data['w_size']), force_points=1.5,\n",
    "            arrowprops=dict(arrowstyle='->', color='red'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Statistics\n",
    "Now we can run the code on the 10 different models and see if the EA is producing consistent outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "num_of_runs = 10\n",
    "def run_multiple(num_of_runs=10):\n",
    "    boxplot_accs = []\n",
    "    boxplot_w_sizes = []\n",
    "    boxplot_fitnesses = []\n",
    "    for i in range(num_of_runs):\n",
    "        # 1. Init model\n",
    "        model = get_model()\n",
    "        # 2. Init variables\n",
    "        base_w_size, base_acc = eval_phenotype(model)\n",
    "        orig_weights = np.array(model.layers[5].weights[0])\n",
    "        orig_biases = np.array(model.layers[5].weights[1])\n",
    "        weights = orig_weights.copy()\n",
    "        model.layers[5].set_weights([weights, orig_biases])\n",
    "        population_size = 20\n",
    "        min_threshold = 1e-6\n",
    "        max_threshold = max(abs(np.amin(orig_weights)), abs(np.amax(orig_weights)))\n",
    "        max_size = weights.shape[0] * weights.shape[1]\n",
    "\n",
    "        population = get_population(population_size, min_threshold, max_threshold)\n",
    "\n",
    "        # 3. Run EA in N epochs\n",
    "        epochs = 30\n",
    "        population = run(population, min_threshold, max_threshold, base_w_size, base_acc, epochs=epochs)\n",
    "        # 4. Save stats\n",
    "        output = list(sorted(population, key=lambda x: x.fitness, reverse=True)[:int(population_size *0.75)])\n",
    "        labels = ['acc', 'w_size', 'fitness', 'threshold']\n",
    "        data = []\n",
    "\n",
    "        for ind in output:\n",
    "            if ind.w_size > 10:\n",
    "                data.append([ind.acc, max_size/ind.w_size, ind.fitness, ind.threshold])\n",
    "\n",
    "        data = pd.DataFrame(data, columns=labels)\n",
    "        boxplot_accs = boxplot_accs + list(data['acc'])\n",
    "        boxplot_w_sizes = boxplot_w_sizes + list(data['w_size'])\n",
    "        boxplot_fitnesses = boxplot_fitnesses + list(data['fitness'])\n",
    "    return boxplot_accs, boxplot_w_sizes, boxplot_fitnesses\n",
    "\n",
    "boxplot_accs, boxplot_w_sizes, boxplot_fitnesses = run_multiple(num_of_runs=num_of_runs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Boxplot accuracy, Compression and Fitness values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(1, 3, constrained_layout=True)\n",
    "axes[0].boxplot([np.array(boxplot_accs).transpose().flatten()], notch=True)\n",
    "axes[0].set_title('Accuracy')\n",
    "axes[1].boxplot([max_size / np.array(boxplot_w_sizes).transpose().flatten()], notch=True)\n",
    "axes[1].set_title('Compression ratio')\n",
    "axes[2].boxplot([np.array(boxplot_fitnesses).transpose().flatten()], notch=True)\n",
    "axes[2].set_title('Fitness')\n",
    "plt.suptitle(f'Collected values fror {num_of_runs} runs, each of {epochs} epochs.')\n",
    "\n",
    "for ax in axes:\n",
    "    ax.tick_params(bottom=False, labelbottom=False)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "tensorflow/datasets",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}