Continuous_Fermentation_ApproxDynamicProg.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Continuous_Fermentation_ApproxDynamicProg.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "eb7v2xsqd1if",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Author: Max Mowbray, PhD Candidate, University of Manchester\n",
        "# importing relevant python packages and declaring reference notation\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import scipy.integrate as scp\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy.random as rnd\n",
        "from scipy.spatial.distance import cdist\n",
        "from scipy.optimize import minimize\n",
        "eps  = np.finfo(float).eps\n",
        "import csv"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6BKHw1wwdi0",
        "colab_type": "text"
      },
      "source": [
        "The aim of this workbook is to provide a walk through of the programming behind applying reinforcement learning algorithms to discrete problems. \n",
        "\n",
        "Firstly, we will consider a simple continuous fermentation problem. The states defining the system are simply Biomass, $X$ and Nitrate, $N$ concentrations. This dynamic system can be described as:\n",
        "\n",
        "$$\\frac{\\text{d}x}{\\text{d}t}=\\mu~x~\\frac{N}{N+K_N}-\\mu_d~x^2$$ \n",
        "\n",
        "$$\\frac{\\text{d}N}{\\text{d}t}=-Y_{N/x}~\\mu~\\frac{N}{N+K_N}x+F_{N_{in}}$$\n",
        "\n",
        "In this case, the control available to the agent is purely desribed the nitrate inflow rate $$F_{N_{in}}\\in[0,7]$$\n",
        "\n",
        "This means the nitrate inflow rate can take any value in the range [0,7] $mg/h$ (please note units, $X$ $g/L$ and $N$ $mg/L$). However, intially we want to limit the number of state-action pairs for the sake of computational intensity - hence may only allow values that are factors of $0.5$...\n",
        "\n",
        "In the case of tabular implementation, we discretise the system and structure the state space on a grid of 'acceptable' state values. Therefore, we implement a rounding policy to round the 'real' system values to those accepted (the real state values are those observed after integration of the system of ODEs described over one time step). We may introduce stochasticity (randomness) into our system via a stochastic rounding policy (see discrete_env method).\n",
        "\n",
        "However, for an agent to learn we must define some objective function for our system. One such objective function may act to reduce the waste in the system at the end of the run, whilst maximising biomass concentration (or more succinctly, as maximisation of productivity)\n",
        "\n",
        "$$f_{obj_1}=100x_{t_f}-N_{t_f}$$\n",
        "\n",
        "We may explore a range of objective functions and develop complexity as the work progresses\n",
        "\n",
        "So our first steps are to define the system we wish to train our agent with as well as the rounding policy. Here we may make use of python's object-oriented programming paradigm:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OeTioni6-76Z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Model_env: \n",
        "    \n",
        "    # --- initializing model --- #\n",
        "    def __init__(self, parameters, steps, tf, x0, modulus):\n",
        "        \n",
        "        # Object variable definitions\n",
        "        self.parameters, self.steps = parameters, steps\n",
        "        self.x0, self.dt, self.tf      = x0, tf/steps, tf                \n",
        "        self.modulus                   = modulus          # two column array [biomass nitrate ]\n",
        "        \n",
        "    # --- dynamic model definition --- #    \n",
        "    # model takes state and action of previous time step and integrates -- definition of ODE system at time, t\n",
        "    def model(self, t, state, control):\n",
        "        # internal definitions\n",
        "        params = self.parameters\n",
        "        FCn   = control\n",
        "                \n",
        "        # state vector\n",
        "        Cx  = state[0]\n",
        "        Cn  = state[1]\n",
        "        \n",
        "        # parameters\n",
        "        u_m  = params['u_m']; K_N  = params['K_N'];\n",
        "        u_d  = params['u_d']; Y_nx = params['Y_nx'];\n",
        "        \n",
        "        # algebraic equations\n",
        "        \n",
        "        # variable rate equations\n",
        "        dev_Cx  = u_m * Cx * Cn/(Cn+K_N) - u_d*Cx**2\n",
        "        dev_Cn  = - Y_nx * u_m * Cx * Cn/(Cn+K_N) + FCn\n",
        "        \n",
        "        return np.array([dev_Cx, dev_Cn],dtype='float64')\n",
        "    \n",
        "    def discrete_env(self, state):\n",
        "        # discretisation of the system, with introduction of stochasticity in terms of modulus\n",
        "        modulus = self.modulus\n",
        "        \n",
        "            \n",
        "        resid = state % modulus\n",
        "        resid = resid/modulus\n",
        "        UB = 1 - resid\n",
        "        draw =  np.ones(2)\n",
        "\n",
        "        for i in range(state.shape[0]):\n",
        "            if draw[i] < UB[i]:\n",
        "              state[i] = state[i] - resid[i] * modulus[i]\n",
        "            else:\n",
        "              state[i] = state[i] - resid[i] * modulus[i] + modulus[i]\n",
        "        \n",
        "        # fixes for representation \n",
        "        # Nitrate \n",
        "        if state[1] < 0:\n",
        "          state[1] = 0\n",
        "        elif state[1] < self.modulus[1]/2:\n",
        "          state[1] = 0\n",
        "        \n",
        "        # Biomass\n",
        "        f = str(self.modulus[0])\n",
        "        decimal = f[::-1].find('.')  \n",
        "        state[0] = np.round(state[0], decimal)\n",
        "        \n",
        "        return state\n",
        "\n",
        "    def reward(self, state):\n",
        "      reward = 100*state[-1][0] - state[-1][1]              # objective function 1\n",
        "      return reward"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K_IhEDCMAU73",
        "colab_type": "text"
      },
      "source": [
        "Note the syntax of the reward allocation. It is based on the final state values. Therefore, we will have to reallocate the reward to actions taken within a sequence after the run. \n",
        "\n",
        "Next we may define the agent. Conventionally, the agent observes the state ($S_{t}$) and interacts with the environment at time $t$ via some action ($A_{t}$), receives feedback about the quality of the action ($R_{t+1}$) and the next state of the environment ($S_{t+1}$).\n",
        "\n",
        "However, in this case the agent will only receive reward at the terminal time, $T$. Therefore, we must devise an appropriate back allocation policy, in order for the agent to learn the value of each action. This is otherwise known as temporal credit assignment and is a key factor in developing solid reinforcement learning algorithms and ultimately agents. Initially we will try\n",
        "\n",
        "$$R_{t+1}=R_{T}~\\gamma_{1}^2$$\n",
        "$$\\gamma_{1}\\in[0,1]$$\n",
        "\n",
        "Here $\\gamma_{1}$ is some discount factor that determines how we value actions later in the sequence relative to actions earlier in the sequence. This may be a parameter for investigation during the training of an efficacious agent.\n",
        "\n",
        "The agent defined below is based on Monte Carlo Learning, which relies on large state space sweeps, experience of playing actions and updating the expected return $G_{t}$ from that state-action, ($s,a$) pair.\n",
        "\n",
        "We may define the expected return from a ($s,a$) pair recursively as:\n",
        "\n",
        "$$G_{t} = R_{t+1} + \\gamma_{2}~R_{t+2}+ \\gamma_{2}^{2}~R_{t+3} + ... +\\gamma_{2}^{T-1}R_{T}$$\n",
        "$$$$\n",
        "$$G_{t} = R_{t+1} + \\gamma_{2}~G_{t+1}$$\n",
        "\n",
        "$$$$\n",
        "$$\\gamma_{2} \\in [0,1] $$\n",
        "\n",
        "Where $\\gamma_{2}$ is a discount factor that signifies how an agent values immediate reward over future return. Note the state-action value represents the expected return $G_{t}$ from choosing an action, $A_{t}$ in a given state, $S_{t}$.\n",
        "\n",
        "Make sure to understand the initialisation of the $(s,a)$ dictionary as well as the count (under the _init_ method). Later, we will replace the $(s,a)$ dictionary with a function approximator (such as neural network).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OE5MGTDuA3qv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "############# Defining Agent To Be Trained ##################\n",
        "class greedye_MCL:\n",
        "    def __init__(self, num_actions, modulus, state_UB, disc1, disc2, movements):\n",
        "      self.name = \"e-Greedy\"\n",
        "      # defining number of actions available to agent, the line distance of the state space, the upper bound of the state space and the number of s-a pairs observed per sequence (from t = 0...T)\n",
        "      self.num_actions, self.modulus, self.UB, self.movements  = num_actions, modulus, state_UB, movements    \n",
        "      self.disc1, self.disc2 = disc1, disc2                                                                       # defining discount factors\n",
        "      self.X_s = np.linspace(0, int(self.UB[0]), int(self.UB[0]/self.modulus[0] + 1), dtype = np.float64)\n",
        "      self.N_s = np.linspace(0, int(self.UB[1]), int(self.UB[1]/self.modulus[1] + 1), dtype = np.float64)\n",
        "      f1, f2   = str(self.modulus[0]), str(self.modulus[1])\n",
        "      decimal1, decimal2 = f1[::-1].find('.'), f2[::-1].find('.')                                           \n",
        "      # defining state-action value dictionary based on grid definition from X_s and N_s\n",
        "      self.d = {(X_si, N_si, ti): np.random.randn(num_actions) for X_si in np.round(self.X_s,decimal1) for N_si in np.round(self.N_s,decimal2) for ti in np.arange(1,self.movements+1, dtype =\"int\")}\n",
        "      # initialising count for observation of each state-action pair\n",
        "      self.dcount = {(X_si, N_si, ti): np.zeros(num_actions) for X_si in np.round(self.X_s,decimal1) for N_si in np.round(self.N_s,decimal2) for ti in np.arange(1,self.movements+1, dtype =\"int\")}\n",
        "     \n",
        "    def act(self,state, eps_prob, s):                                      \n",
        "      # e-greedy definition\n",
        "      self.eps_prob = eps_prob\n",
        "      time_to_term  = int(self.movements - s)                          \n",
        "      if np.random.uniform(0,1) <= self.eps_prob:\n",
        "        action = np.random.randint(0,self.num_actions)\n",
        "      else: action = np.argmax(self.d[(state[0],state[1], time_to_term)])\n",
        "      return action\n",
        "\n",
        "    def Learn(self, state, action, reward):\n",
        "        self.reward = reward\n",
        "        XT, NT      = state[-1,0], state[-1,1] \n",
        "        # takes some state and attributes discounted reward to action via QL update \n",
        "        for i in range(0,action.shape[0]):                 # attributing value to actions\n",
        "          indx         = action[i]\n",
        "          time_to_term = int(self.movements - i)           # finding time index\n",
        "          Gt = 0                                           # return at time t = 0\n",
        "          W = 1\n",
        "          self.dcount[(state[i,0],state[i,1],time_to_term)][int(indx)] = self.dcount[(state[i,0],state[i,1],time_to_term)][int(indx)] + 1      # updatingcount\n",
        "          if i < action.shape[0]-1:\n",
        "              # finding return from time,t=i in sequence\n",
        "              for j in range(i, action.shape[0]):\n",
        "                  time_to_ter = int(self.movements - j)                                         # defining time to termination (time index)\n",
        "                  Rtp1 = (reward * self.disc1**(time_to_ter))                                   # allocating reward based on some backallocation function discounted to time and implementation of variational derivative\n",
        "                  Gt += Rtp1*self.disc2**(j-i)                                                  # updating observed return and discouting via \\gamma_{2}\n",
        "          else:\n",
        "              Rtp1 = (reward * self.disc1**(time_to_term)) \n",
        "              Gt += Rtp1                                                                        # for last action in sequence expected return is equivalent to reward observed\n",
        "          alpha = W / self.dcount[(state[i,0],state[i,1],time_to_term)][int(indx)]              # updating learning parameter based on number s-a pair count\n",
        "          self.d[(state[i,0],state[i,1],time_to_term)][int(indx)] = self.d[(state[i,0],state[i,1], time_to_term)][int(indx)] * (1-alpha) + alpha * (Gt)          \n",
        "          return   \n",
        "          \n",
        "    def learned(self):\n",
        "      dictionary = self.d\n",
        "      return dictionary \n",
        "  \n",
        "############ Defining learned Agent #################\n",
        "class Greedye_MCLearned:\n",
        "  # defining trained agent\n",
        "    def __init__(self, num_actions, d, movements):\n",
        "      self.name = \"e-Greedy\"\n",
        "      self.num_actions = num_actions\n",
        "      self.d, self.movements = d, movements\n",
        "     \n",
        "    def act(self,state, eps_prob, s):                                      \n",
        "      #e-greedy definition\n",
        "      self.eps_prob = eps_prob\n",
        "      time_to_term = int(self.movements - s)  \n",
        "      if np.random.uniform(0,1) <= self.eps_prob:\n",
        "        action = np.random.randint(0,self.num_actions)\n",
        "      else: action = np.argmax(self.d[(state[0],state[1], time_to_term)])\n",
        "      return action"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7gTWyAJHTfoq",
        "colab_type": "text"
      },
      "source": [
        "Next we look to define the Experiment. This is where we enable interaction between our previously defined agent and environment and can be imagined as some interface between the two. The main bulk of the interesting stuff happens within the 'simulation' method. Essentially, we allow the agent to interact with the environment a defined number of times within a run. We enable our agent to select a control action under a given policy of behaviour and then simulate the dynamic change of the environment. We follow this process across an entire run and then repeat this many times. Each time we complete a run, we complete one epoch of training. Typically, we require the agent to experience the environment many, many times before converging upon an optimal policy of behaviour (i.e. 1e6 epochs).\n",
        "\n",
        "In this workbook, we have defined the policy of behaviour within the experiment, but this could well be defined as one of the class methods/functions of the agent as detailed above. The policy of behaviour is defined in the experimental design section (the next section).\n",
        "\n",
        "I strongly advise trying to disect the code here. It will help you understand how the agent and environment interact and how we evaluate the quality of the agent's actions within a sequence."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CPrjC8p2UV7e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "################# --- Training Agent --- #####################\n",
        "class Experiment(object):\n",
        "    def __init__(self, env, agent, controls, episodes,xi):\n",
        "      self.env , self.agent,                     = env, agent \n",
        "      self.controls, self.episodes, self.xi      = controls, episodes, xi\n",
        "\n",
        "    def eps_prob(self,ei,episodes):\n",
        "        if self.xi == int(1):\n",
        "            F = 0.1\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(2):\n",
        "            F = 0.2\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(3):\n",
        "            F = 0.3\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(4):\n",
        "            F = 0.4\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(5):\n",
        "            F = 0.5\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(6):\n",
        "            F = 0.05\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        elif self.xi == int(7):\n",
        "            F = 0.01\n",
        "            G = -np.log(0.1)*F*episodes     # =no of episodes until behave =0.1\n",
        "            if ei < G:\n",
        "                behave = np.exp(-ei/(episodes*F))\n",
        "            else:\n",
        "                behave = 0.1\n",
        "        else: behave = 1                    # behave randomly all the time\n",
        "        return behave      \n",
        "            \n",
        "       \n",
        "    def simulation(self):\n",
        "      # Simulation takes environment, imparts control action from e-greedy policy and simulates, observes next state to the end of the sequence and outputs reward\n",
        "      # internal definitions\n",
        "      discrete_env        = self.env.discrete_env\n",
        "      dt, movements, x0   = self.env.dt, int(self.env.tf/float(self.env.dt)), self.env.x0\n",
        "      model, ctrls        = self.env.model, self.controls       #takes set of control options\n",
        "      episodes            = self.episodes\n",
        "\n",
        "      # compile state and control trajectories\n",
        "      xt      = np.zeros((movements+1, x0.shape[0], episodes))\n",
        "      tt      = np.zeros((movements+1))\n",
        "      c_hist  = np.zeros((movements, episodes))\n",
        "      ctrl    = np.zeros((movements, episodes))\n",
        "      reward  = np.zeros((episodes))\n",
        "\n",
        "      for ei in range(episodes):\n",
        "        # initialize simulation\n",
        "        current_state = x0\n",
        "        xt[0,:,ei]    = current_state\n",
        "        tt[0]         = 0.\n",
        "        \n",
        "        \n",
        "        # define e greedy policy exploration\n",
        "        eps_prob = self.eps_prob(ei,episodes)\n",
        "      \n",
        "        # simulation\n",
        "        for s in range(movements):\n",
        "            action_indx  = self.agent.act(current_state, eps_prob, s)        # select control for this step from that possible\n",
        "            ctrl[s,ei]   =  ctrls[action_indx]                               # find control action relevant to index from agent.act\n",
        "            c_hist[s,ei] = action_indx                                       # storing control history for each epoch\n",
        "            ode          = scp.ode(self.env.model)                           # define ode\n",
        "            ode.set_integrator('lsoda', nsteps=3000)                         # define integrator\n",
        "            ode.set_initial_value(current_state,dt)                          # set initial value\n",
        "            ode.set_f_params(ctrl[s,ei])                                     # set control action\n",
        "            current_state = list(ode.integrate(ode.t + dt))                  # integrate system\n",
        "            current_state = discrete_env(np.array(current_state))\n",
        "            xt[s+1,:,ei]  = current_state                                    # add current state Note: here we can add randomnes as: + RandomNormal noise\n",
        "            tt[s+1]       = (s+1)*dt\n",
        "        \n",
        "        for i in [0, 0.2, 0.4, 0.6, 0.8]:\n",
        "            if i == ei/episodes:\n",
        "                print('Simulation is', i*100 , ' percent complete')\n",
        "\n",
        "        reward[ei] = self.env.reward(xt[:,:,ei])\n",
        "        self.agent.Learn(xt[:,:,ei], c_hist[:,ei], reward[ei])\n",
        "            \n",
        "      d = self.agent.learned()\n",
        "      return reward, d \n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V0hyCmHPYrXP",
        "colab_type": "text"
      },
      "source": [
        "We define a similar experimental environment to validate our agent. Please try to define the differences between the training and validation experiments for your understanding."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DOMqYQ_lY3fl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#################### --- Validation Simulation --- #####################\n",
        "class Experiment_Done(object):\n",
        "    def __init__(self, env, agent, controls, episodes, eps):\n",
        "      self.env , self.agent,         = env, agent \n",
        "      self.controls, self.episodes   = controls, episodes\n",
        "      self.eps                       =  eps\n",
        "       \n",
        "    def simulation(self):\n",
        "      # Simulation takes environment and simulates, next iteration and outputs reward\n",
        "      # internal definitions\n",
        "      discrete_env        = self.env.discrete_env\n",
        "      dt, movements, x0   = self.env.dt, int(self.env.tf/float(self.env.dt)), self.env.x0\n",
        "      model, ctrls        = self.env.model, self.controls       #takes set of control options\n",
        "      episodes            = self.episodes\n",
        "\n",
        "      # compile state and control trajectories\n",
        "      xt        = np.zeros((movements+1, x0.shape[0], episodes))\n",
        "      tt        = np.zeros((movements+1))\n",
        "      c_hist    = np.zeros((movements, episodes))\n",
        "      ctrl      = np.zeros((movements, episodes))\n",
        "      reward    = np.zeros((episodes))\n",
        "      plot_m    = np.array([episodes-1])\n",
        "\n",
        "      for ei in range(episodes):\n",
        "        # initialize simulation\n",
        "        current_state = x0\n",
        "        xt[0,:,ei]    = current_state\n",
        "        tt[0]         = 0.\n",
        "\n",
        "        # define e greedy policy exploration\n",
        "        eps_prob = self.eps        #act greedily\n",
        "      \n",
        "        # simulation\n",
        "        for s in range(movements):\n",
        "            action_indx   = self.agent.act(current_state, eps_prob, s)        # select control for this step from that possible\n",
        "            ctrl[s,ei]    =  ctrls[action_indx]                              # find control action relevant to index from agent.act\n",
        "            c_hist[s,ei]  = action_indx                                      # storing control history for each epoch\n",
        "            ode           = scp.ode(self.env.model)                             # define ode\n",
        "            ode.set_integrator('lsoda', nsteps=3000)                        # define integrator\n",
        "            ode.set_initial_value(current_state,dt)                         # set initial value\n",
        "            ode.set_f_params(ctrl[s,ei])                                    # set control action\n",
        "            current_state = list(ode.integrate(ode.t + dt))                 # integrate system\n",
        "            current_state = discrete_env(np.array(current_state))\n",
        "            xt[s+1,:,ei]  = current_state                                # add current state Note: here we can add randomnes as: + RandomNormal noise\n",
        "            tt[s+1]       = (s+1)*dt\n",
        "        \n",
        "        \n",
        "        reward[ei] = self.env.reward(xt[:,:,ei])\n",
        "        \n",
        "      return reward  \n",
        "      \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uvUfHQA-U68S",
        "colab_type": "text"
      },
      "source": [
        "Now we have defined, both the experiment, environment and the interface between the two. All we have left to do is design/define and run the experiment in order to train our agent.\n",
        "\n",
        "This is done as follows. Again, please try to understand the interactions within the code and how it works. Feel free to send me a message if you have any questions :)\n",
        "\n",
        "I suggest having a go at defining both a Q Learning agent and SARSA agent and trying to run them both. Then you will gain hands on experience of coding the algorithms and understanding how the code works."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TqCIdMVyV9Qz",
        "colab_type": "code",
        "outputId": "fbc4f7cf-67cc-4e76-95f1-54e92b2e4fc5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "####################### -- Defining Parameters and Designing Experiement -- #########################\n",
        "#investigating steps_, alpha and discount factor for 1,000,000 epochs of training \n",
        "# Model definitions: parameters, steps, tf, x0  \n",
        "      \n",
        "p        = {'u_m' : 0.0923*0.62, 'K_N' : 393.10, 'u_d' : 0.01, 'Y_nx' : 504.49}                 # model parameter definitions\n",
        "steps_   = np.array([10])                                                                       # number of control interactions\n",
        "tf       = 16.*24                                                 \n",
        "x0       = np.array([0.5,150.0])                                                                # initial conditions of environment\n",
        "modulus  = np.array([0.05, 10])                                                                 # line distance of state space\n",
        "state_UB = np.array([5, 1000])                                                                  # state space upper bound\n",
        "\n",
        "# Agent definitions: num_actions, eps_prob, alpha, discount\n",
        "agent_name = 'MonteCarlo'\n",
        "num_actions = 15                                                                                # number (range) of actions available to agent\n",
        "\n",
        "disc1 = np.array([0.85])                                                                        # discount factor in back-allocation\n",
        "disc2 = np.array([0.95])                                                                        # discount factor in agent learning\n",
        "xi_ = np.array([3])                                                                             # Epsilon greedy definition (from experiment)\n",
        "\n",
        "# Experiment defintions: env, agent, controls, episodes\n",
        "controls          = np.linspace(0,7,num_actions)                                                         # defining range of controls\n",
        "episodes_train    = 10000                                                                          # number of training epochs\n",
        "episodes_valid    = 1000                                                                           # numeber of validation epochs\n",
        "reward_training   = np.zeros((episodes_train, xi_.shape[0], disc1.shape[0], disc2.shape[0]))      # memory allocation \n",
        "reward_validation = np.zeros((episodes_valid, xi_.shape[0], disc1.shape[0], disc2.shape[0]))    # memory allocation \n",
        "bracket           = int(1000)\n",
        "\n",
        "## These function are for plotting the output \n",
        "def EpochNoMean(data, bracket):\n",
        "  nrows           = int(data.shape[0]/bracket)\n",
        "  plot_prep_mean  = np.zeros((int(nrows)))\n",
        "  for f in range(0,nrows):\n",
        "    x = data[f*bracket:f*bracket+ bracket-1]\n",
        "    y = np.mean(x,0)\n",
        "    plot_prep_mean[f] = y\n",
        "  return plot_prep_mean\n",
        "\n",
        "#plot of 1000 epoch mean throughout training\n",
        "def Plotting(data, nrows, bracket, pNo_mean, agent_name):\n",
        "  plt.figure(figsize =(20,16))\n",
        "  plt.scatter(np.linspace(0,len(data),nrows), data, label= 'Mean R over 1000 epochs')\n",
        "  plt.xlabel('Training epochs (1e3)',  fontsize=28)\n",
        "  plt.ylabel('Mean reward over ' + str(bracket)+ ' epochs', fontsize=28)\n",
        "  plt.tick_params(labelsize=24)\n",
        "  plt.savefig('insert your computer path here' + str(pNo_mean) + '_' + str(agent_name) +'.png')\n",
        "\n",
        "# running experiement\n",
        "env                 = Model_env(p, steps_, tf, x0, modulus)                                               # calling environment\n",
        "agent               = greedye_MCL(num_actions, modulus, state_UB, disc1, disc2, steps_)                 # calling agent\n",
        "experiment          = Experiment(env, agent, controls, episodes_train, xi_)                        # calling training experiment\n",
        "reward_training, d  = experiment.simulation()                                              # running training experiment\n",
        "agent               = Greedye_MCLearned(num_actions, d, steps_)                                         # calling learned agent\n",
        "exp_done            = Experiment_Done(env, agent, controls, episodes_valid, 0)                       # calling validation experiment\n",
        "reward_validation   = exp_done.simulation()                                                 # running validation experiment\n",
        "reward_train_mean   = EpochNoMean(reward_training,bracket)\n",
        "nrows               = int(len(reward_train_mean))\n",
        "x_o                 = Plotting(reward_train_mean, nrows, bracket, \"Time_allocation\", agent_name )\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Simulation is 0  percent complete\n",
            "Simulation is 20.0  percent complete\n",
            "Simulation is 40.0  percent complete\n",
            "Simulation is 60.0  percent complete\n",
            "Simulation is 80.0  percent complete\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABLAAAAOzCAYAAACoEv/vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzde5RmV1kn/u9Dp0OaBNIEAyFNQsAZ\nwiAwE2lE7nJt7gQQRHEGogIu8cLSFTWjM+uno4ATLyOijKhAVBABQwsTmEbQCCqICS0GQnqiXBI6\ngAHSCYFK0iTP74/3FOmU3V1Vb1V3ner381nrrF3vOXuf/fTKf9/svU91dwAAAABgrG631gUAAAAA\nwMEIsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAUTtqrQtYj77lW76lTzvttLUu\nAwAAAOCIcfHFF3+pu0/c3zMB1hROO+20XHTRRWtdBgAAAMARo6o+e6Bn62YLYVWdWlUvr6p3VdUV\nVXVjVX21qj5WVa+qqrsv4R2bq+q/VdVFVXVNVX29qj5VVedX1YsOwz8DAAAAgGVaFyuwquqUJJ9J\nUvvcvi7JsUkeOFwvqarndPdfHeAdj0rytiR3HW7dOFz3Gq4HJnnjISgfAAAAgBVYLyuwNgztBUme\nm+SE7j4+yR2SPCXJp5PcOcn2qjpp4eCq+vYk784kvHpnkgd19zHDOzYneVKSNx/yfwUAAAAAy7Yu\nVmAluSbJGd39sX1vdvdNSd5TVU9JsjPJnZK8NMkvzPepqg1J3pDJaq03JfnP3d37vOPaJDuGCwAA\nAICRWRcrsLr72oXh1YLnlyX58PDzQQsePy2T7YFzSX583/AKAAAAgPFbFwHWEn15aDcsuP+Cod3R\n3V85jPUAAAAAsAqOiACrqo5K8vDh58cXPH7o0O6sqi1V9bqq2j18xfDKqvqjqnrA4asWAAAAgOU4\nIgKsJC9LclKSW5KcN3+zqo5Jco/h552T/GOSFye5SyZbCu+R5PuTXFxVzz+cBQMAAACwNOs+wKqq\nByZ55fDzNd196T6PN+/z948nOTrJ85Mc192bkzwgyd8n2ZjkDVV1n4PM85KquqiqLrr66qtX9d8A\nAAAAwIGt6wCrqu6eZHuSTUkuTvIzC7rcbsHfP9ndf9rd30iS7v54kmcmuT7JMUlefqC5uvt13b21\nu7eeeOKJq/ivAAAAAOBg1m2AVVUnJHlvknsluTzJU7v7hgXdrt/n72uTvHHhe7r7i0nePPx83OpX\nCgAAAMBKrMsAq6qOT7Ijyf2TXJHk8UMQtdBXc2uI9S/dffMBXrlraE9Z1UIBAAAAWLF1F2BV1bFJ\n3p1ka5IvZBJeXbG/vt3dST6xjNf3yisEAAAAYDWtqwCrqjYleVeShyX5cibh1eWLDHvf0H5rVW04\nQJ/7Du1nVlwkAAAAAKtq3QRYVXV0kvOTPCbJniRP7O6lrK56U5Jbkhyf5Kz9vPduSb5v+Pnu1akW\nAAAAgNWyLgKsYeXUm5M8KZNzrZ7c3R9dytju/mSSPxh+/lpVPa+qjhre+22ZfMXw2CTXJPmN1a4d\nAAAAgJU5aq0LWKKHJ3nO8PfGJNur6kB9r+zuBy+49+NJvjXJY5P8aZIbqurGTFZlJZMvFD6nu69a\n1aoBAAAAWLH1EmDtu1LsmOE6kBsW3ujuG6rqCUlenOSFSe43vOOfk7wnybndfeXqlQsAAADAalkX\nAVZ3X5jkgEuulviOW5L87nABAAAAsE6sizOwAAAAAJhdAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUB\nFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAA\nABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAAAIBRO2qtCwAAAABgebbv3J1zd+zKVXvmcvLm\nTTl72+k584wta13WISPAAgAAAFhHtu/cnXPOvyRze29OkuzeM5dzzr8kSY7YEMsWQgAAAIB15Nwd\nu74ZXs2b23tzzt2xa40qOvQEWAAAAADryFV75pZ1/0ggwAIAAABYR07evGlZ948EAiwAAACAdeTs\nbadn08YNt7m3aeOGnL3t9DWq6NBziDsAAADAOjJ/ULuvEAIAAAAwWmeeseWIDqwWsoUQAAAAgFET\nYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAA\nAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIya\nAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAA\nAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDU\nBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIA\nAABg1ARYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACj\nJsACAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYA\nAAAAoybAAgAAAGDUBFgAAAAAjNpRa10AAAAAcHht37k75+7Ylav2zOXkzZty9rbTc+YZW9a6LDgg\nARYAAADMkO07d+ec8y/J3N6bkyS798zlnPMvSRIhFqNlCyEAAADMkHN37PpmeDVvbu/NOXfHrjWq\nCBYnwAIAAIAZctWeuWXdhzEQYAEAAMAMOXnzpmXdhzEQYAEAAMAMOXvb6dm0ccNt7m3auCFnbzt9\njSqCxTnEHQAAAGbI/EHtvkLIeiLAAgAAgBlz5hlbBFasK7YQAgAAADBqAiwAAAAARk2ABQAAAMCo\nCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFFbNwFWVZ1aVS+vqndV1RVVdWNV\nfbWqPlZVr6qqux9g3GlV1Uu4th7ufxMAAAAAiztqrQtYiqo6JclnktQ+t69LcmySBw7XS6rqOd39\nVwd51RcP8mzvSusEAAAAYPWtiwAryYahvSDJG5O8v7uvqaqjkzwuyW8nuVeS7VV1end/YX8v6e6T\nDkexAAAAAKye9bKF8JokZ3T307r77d19TZJ0903d/Z4kT0lyQ5I7JXnpGtYJAAAAwCpbFwFWd1/b\n3R87yPPLknx4+Pmgw1MVAAAAAIfDetlCuBRfHtoNB+0FAMC6t33n7py7Y1eu2jOXkzdvytnbTs+Z\nZ2xZ67IAgENkXazAWkxVHZXk4cPPjx+k34eq6rqqmquqT1fVH1fVIw5PlQAArIbtO3fnnPMvye49\nc+kku/fM5ZzzL8n2nbvXujQA4BA5IgKsJC9LclKSW5Kcd5B+3zn0SZLTkrwgyQer6n9VVR1wFAAA\no3Hujl2Z23vzbe7N7b055+7YtUYVAQCH2roPsKrqgUleOfx8TXdfuqDLDUl+J8mjktyxuzcnuUMm\nZ2W9a+jzE0nOWWSel1TVRVV10dVXX71q9QMAsDxX7Zlb1n0AYP1b1wFWVd09yfYkm5JcnORnFvbp\n7i9098u6+4Pdff1wr7v7o939jCRvG7r+16rafKC5uvt13b21u7eeeOKJq/+PAQBgSU7evGlZ9wGA\n9W/dBlhVdUKS9ya5V5LLkzy1u2+Y4lXzodexSR63SuUBAHCInL3t9GzaeNvv9mzauCFnbzt9jSoC\nAA61dfkVwqo6PsmOJPdPckWSx3f3F6d5V3d/uqquTnJiknuvXpUAABwK818b9BVCAJgd6y7Aqqpj\nk7w7ydYkX8gkvLpibasCAOBwOvOMLQIrAJgh62oLYVVtyuTg9Ycl+XIm4dXlK3znvTJZfZUkn15Z\nhQAAAACstnUTYFXV0UnOT/KYJHuSPLG7P7GEcbVIl1cM7VySv1xRkQAAAACsunURYFXVhiRvTvKk\nJF9N8uTu/ugSh19YVedU1f2H96QmzqiqdyR5/tDvV7r7K6tePAAAAAArsl7OwHp4kucMf29Msv0g\nC6uu7O4H7/P7npmssnpFkr1VdV2SOyTZ9zvLv5XkF1e1YgAAAABWxXoJsPZdKXbMcB3IDQt+n53k\nCUm+I8lJSU5IclOSXUn+NsnruvvvV69UAAAAAFbTugiwuvvCJIudZXWgsW9L8rZVLQgAAACAw2Zd\nnIEFAAAAwOwSYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgA\nAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg\n1ARYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARu2otS4AAACA9Wf7zt05d8euXLVnLidv3pSz\nt52eM8/YstZlAUcoARYAAADLsn3n7pxz/iWZ23tzkmT3nrmcc/4lSSLEAg4JWwgBAABYlnN37Ppm\neDVvbu/NOXfHrjWqCDjSCbAAAABYlqv2zC3rPsBKCbAAAABYlpM3b1rWfYCVEmABAACH3fadu/Pw\nV/1l7vWzF+Thr/rLbN+5e61LYhnO3nZ6Nm3ccJt7mzZuyNnbTl+jioAjnUPcAQCAw8oB4Ovf/H8n\nXyEEDhcBFgAAcFgd7ABwAcj6ceYZW/z3Ag4bWwgBAIDDygHgACyXAAsAADisHAAOwHIJsAAAgMPK\nAeAALJczsAAAgMPKAeAALJcACwAAOOwcAA7ActhCCAAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAA\nYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybA\nAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAA\nAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUB\nFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAA\nABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJ\nsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNgAQAAADBqhyzAqqrbV9Vdq0pI\nBgAAAMDUlh0uVdWxVfXE4Tp2P8/vUlVvT3Jdks8n+UpV/c+q2rgK9QIAAAAwY46aYsyzk5yX5Kok\n99z3QVVVkguSPDhJDbfvlOSnkpyS5HunrhQAAACAmTRNgLVtaLd3980Lnn13ku9I0kk+nuSDSR6Z\n5P5JnldVr+/uv5i2WAAYi+07d+fcHbty1Z65nLx5U87ednrOPGPLWpcFAABHpGkCrAdkElD97X6e\n/Zeh/ViSh3T33qq6/dD3jOG5AAuAdW37zt055/xLMrd38v9xdu+ZyznnX5IkQiwAADgEpjlg/a5D\n+6l9b1bVhiTflUm49TvdvTdJuvvGJK/NZEvhQ6auFABG4twdu74ZXs2b23tzzt2xa40qAgCAI9s0\nAdYJQ3vDgvtnJJk/1P09C55dNrQnTzEfAIzKVXvmlnUfAABYmWkCrBuH9sQF9x85tJ/t7t0Lnn1t\naDdMMR8AjMrJmzct6z4AALAy0wRYnx7ahy64//RMtg9+YD9j7jK0/zrFfAAwKmdvOz2bNt72/8ls\n2rghZ287fY0qAgCAI9s0AdZfZnKe1cuq6n5JUlXPSPLo4fkF+xlz/6H9/BTzAcConHnGlrzy2Q/I\nls2bUkm2bN6UVz77AQ5wBwCAQ2SarxC+JskPZ7KF8JKquibJnTMJtT6bZPt+xmzLZHXWP01ZJwCM\nyplnbBFYAQDAYbLsFVjd/S9Jvj/J1zMJrU4Y2muSfO/81wfnVdXdkzxu+PlXK6oWAAAAgJkzzQqs\ndPefVdUHkzw1yUlJrkryzu6+Zj/dH5DkzcPf756qSgAAAABm1lQBVpJ0978mecMS+r03yXunnQcA\nAACA2TbNIe4AAAAAcNgIsAAAAAAYtam3ECZJVZ2Q5GFJ7p3kjkk2LDamu39xyrlOTfLsTA6E/49J\n7pbkpiSfSvKeJL/Z3Z9fxvu2J3nm8PO87n7RNHUBAAAAcGhNFWBV1V2S/HqS70mycZnDlx1gVdUp\nST6TydcO512X5NgkDxyul1TVc7p70S8dVtWZuTW8AgAAAGDElr2FsKrulOQDSb4/ydGZhErLuaYx\nv7LrgiTPTXJCdx+f5A5JnpLk00nunGR7VZ20SP3HJXl1JgHYZVPWAwAAAMBhMs0ZWD+d5D9kEka9\nP8mTk5yYZEN3326xa8o6r0lyRnc/rbvf3t3XJEl339Td78kkxLohyZ2SvHSRd/2PJKck+W9Jvjhl\nPQAAAAAcJtMESs9K0kn+b5IndveO7v5yd/fqlnar7r62uz92kOeXJfnw8PNBB+pXVd+e5MeS/GOS\n317VIgEAAAA4JKYJsE4b2t86lKHVFL48tPs9SL6qbpfkdzP5N/9Id998uAoDAAAAYHrTBFhzQ3vV\nahayElV1VJKHDz8/foBuP5pka5LXd/eHDkthAAAAAKzYNAHW/MHn91jNQlboZUlOSnJLkvMWPqyq\nLUl+KZNVWj9zeEsDAAAAYCWmCbDekMkB7s9b5VqmUlUPTPLK4edruvvS/XR7dZI7JvnZ7v7yfp4D\nAAAAMFLTBFivT/K+JC+oqv+8yvUsS1XdPcn2JJuSXJz9rK6qqqcleXYmh7z/wQrmeklVXVRVF119\n9dXTvgYAAACAZTrqQA+q6tSDjPuJJL+X5I1V9cwkb0ryySRfX2zC7r5iuUUeoL4Tkrw3yb2SXJ7k\nqd19w4I+x2bytcGbMzm4fepD57v7dUlelyRbt24d0+H1AAAAAEe0AwZYST69hPGV5FnDtRS9yJxL\nUlXHJ9mR5P5Jrkjy+O7+4n66/nSSUzMJ2y6vquMWPJ//YuFR+zz72si+rggAAAAw0w62hbCWcC21\n38IxUxtWVb07ky8KfiGT8OpAq7ruObQvTvLV/VyPGJ6/YJ979wwAAAAAo3Gw1VBnHbYqlqiqNiV5\nV5KHZfJFwcd39+VrWxUAAAAAh9IBA6zuPu9wFrKYqjo6yflJHpNkT5IndvcnDjamu1+U5EUHeeeF\nSR6d5LyhLwAAAAAjM81XCA+7qtqQ5M1JnpTJNr8nd/dH17YqAAAAAA6HFR+ofpg8PMlzhr83Jtle\ndcDjtK7s7gcflqoAAAAAOOSWHWAN51A9d/j5nu6+epH+JyZ58vDzT7p773LnzG1Xih0zXAdywxTv\nBwAAAGCkplmB9dwkb0yyO5NtfYvZk+SXk5yc5KYkb1nuhN19YVbhC4b7ee93rfY7AQAAAFhd05yB\n9bShfWt3f2OxzsOKq7dkEkA9c4r5AAAAAJhh0wRYD0rSST6wjDEf3GcsAAAAACzZNAHWyUN75TLG\nfG5ot0wxHwAAAAAzbJoA65ahXc75WfN9p5kPAAAAgBk2TaD0r0N7v2WMme970C8WAgAAAMBC0wRY\nf5/JgewvWsaYszI5N+uiKeYDAAAAYIZNE2C9dWgfWVU/t1jnqvr5JI9cMBYAAAAAlmTZAVZ3n5/k\nHzJZhfWLVfV/qurxVbVpvk9VbaqqJ1TVBUl+IZPVV//Y3W9ZrcIBAAAAmA3LOYh9X89J8rdJTkny\n5OHqqrpueH6nTAKuDO3nkpy5gjoBAAAAmFFTfRWwuz+X5NuTvH24VcO7Ng/X7XJrgPX2JN/e3Veu\nrFQAAAAAZtG0K7DS3V9O8ryq+ndJnppJoHXi8PhLSS5OckF3//OKqwQAAABgZk0dYM0bAqrfXIVa\nAAAAAODfmGoLIQAAAAAcLitegZUkVbUxyX2SnDDc+kqS/9fde1fj/QAAAADMrhUFWFX1tCQ/luSR\nSW6/4PGNVfXBJK/u7gtWMg8AAAAAs2uqLYRVdXRVvSXJnyd5fJJjMvnq4L7XMcOzd1bVn1TV0atT\nMgAAAACzZNoVWG9K8uxMgqpvJHlfkg8n+cLw/KQkD0nyhGGO52USln3PSooFAAAAYPYsO8Cqqicn\neU6STvLBJC/s7s8coO9pSd6Q5NFJvruqtnX3jmmLBQAAAGD2TLOF8Kyh/XiSJxwovEqS4dm2JJcM\nt35wivkAAAAAmGHTBFjfmcnqq1/r7psW6zz0+dVMtht+5xTzAQAAADDDpgmwThzajy9jzCcWjAUA\nAACAJZkmwJob2uOXMeZOC8YCAAAAwJJME2D9y9A+axljzhzaf55iPgAAAABm2DQB1gWZnGf1w1X1\n9MU6V9WTkvxIJudmXTDFfAAAAADMsGkCrN9Mck2SDUneUVWvr6qHV9XR8x2q6uiqelhV/X6SdyU5\nahjz6tUoGgAAAIDZcdRyB3T3NVX13UneneT2SV44XLdU1XWZrLQ6PreGY5XkxiTf3d3XrErVAAAA\nAMyMaVZgpbv/KslDknwkk4CqMlmRdeckJwx/z9//SJLv6O4LV6FeAAAAAGbMsldgzevuf0rynVW1\nNckTktw/k/AqSb6S5ONJ/qK7L1pxlQAAAADMrKkDrHlDQCWkAgAAAOCQmGoLIQAAAAAcLitegTWv\nqu6d5C7Dzy8n+XR392q9HwAAAIDZtKIAq6oel+RHkzw2yXELHn+tqt6f5Le7+30rmQcAAACA2TXV\nFsKqun1VvTnJe5M8I8kdc+tXB+ev44ZnO6rqLVV1zOqUDAAAAMAsWfYKrKqqJO9M8vhMgqqbk1yY\n5CNJvjh0u1uSByd5TJINSZ6byRcKn7jiigEAAACYKdNsIfyBJE9I0pkEVz/Y3Z/eX8eqOi3J72ey\nxfBxVfWD3f0HU1UKAAAAwEyaZgvhWUP7sSRPPFB4lSTd/ZkkT07yj5ms1jrrQH0BAAAAYH+mCbC+\nLZPVV7/W3d9YrHN3703yq8PP+08xHwAAAAAzbJoAa8PQXrqMMZetYD4AAAAAZtg0Z2B9JpNVWCcs\nY8zmfcYCzLztO3fn3B27ctWeuZy8eVPO3nZ6zjxjy1qXBQAAMErTrIj6s0zOs3ruMsY8L5Nth+dP\nMR/AEWX7zt055/xLsnvPXDrJ7j1zOef8S7J95+61Lg0AAGCUpgmwfj3JvyT5wap6wWKdq+r7kvxQ\nkk8NYwFm2rk7dmVu7823uTe39+acu2PXGlUEAAAwbssOsLr7q0ken+SjSf6wqt5ZVc+qqi1VtbGq\njhr+flZV/XmSP0pyUZLHd/d1q1s+wPpz1Z65Zd0HAACYdcs+A6uq9l02UEmeOlwHHJLkwUk+VVUH\n6tPdPc15XADrzsmbN2X3fsKqkzdvWoNqAAAAxm+aLYS1z7Xw9/6upfQ5YLIFcKQ5e9vp2bRxw23u\nbdq4IWdvO32NKgIAABi3aVY9/cKqVwEwQ+a/NugrhAAAAEtT3b3WNaw7W7du7YsuumitywAAAAA4\nYlTVxd29dX/PptlCCAAAAACHjQALAAAAgFFb8Zf/qupbkzw0yUlJ7pDkd7r7Syt9LwAAAAAkKwiw\nquo/JfnNJI9Y8OjtSb60T78fTfLfk1yb5H7dvXfaOQEAAACYPVNtIayqJyf5u0zCq9rn2p8/zGRl\n1r2TPG2a+QAAAACYXcsOsKrqbknekuSYJJcleUqSOx2of3dfl+Rdw88nT1EjAAAAADNsmhVYL09y\nxyRXJnlEd//f7r5+kTEXZrJC60FTzAcAAADADJsmwHpSkk7y6919zRLHfHJoT5tiPgAAAABm2DQB\n1r2G9sPLGHPt0N5xivkAAAAAmGHTBFhHTzFmPrj62hRjAQAAAJhh0wRYXxzaey9jzPzZV7unmA8A\nAACAGTZNgPWhoX36UjpX1YYkL87k3KwPTDEfAAAAADNsmgDrjzP5ouDzquohB+tYVZXkt5Pcb7j1\nxinmAwAAAGCGLTvA6u53J3nvMPa9VfWTVXXqPl2Orap7V9V/SfKR3Lr66k+7+yOrUTQAAAAAs+Oo\nKcc9L8n7Mznb6tzh6uHZwq8TVpK/S/JDU84FAAAAwAybZgthuvu6JA9L8stJrsskpNrf9fUkr0ry\nmO7++moUDAAAAMBsmXYFVrp7b5L/VlWvSvLoJFuT3DXJhiRfSrIzyfu7+9rVKBQAAACA2TR1gDWv\nu7+W5N3DBQAAAACraqothAAAAABwuAiwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAA\nAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYtaNWMriqNiY5LsmmJHNJru/uvatRGAAA\nAAAkywywquqMJM9K8ogk90ty4n76XJ3k0iR/k+Qd3b1zFeoEAAAAYEYtKcCqqvsn+Y0kj9339gG6\n3zWTYOvRSX6uqv4yycu7+xMrKRQAAACA2bRogFVV25K8LcmxuTW0+nqSf0lyZZLrk9yY5PaZbCc8\nJcm3JrnD0PexST5UVc/p7r9Y1eoBAAAAOOIdNMCqqi1J/iSTYOqWJG9I8vokH+nubxxk3FFJHpzk\nB5O8cBj/lqp6QHdftUq1AwAAADADFvsK4Y8l2ZzJiqvHdvcPdfffHSy8SpLu/kZ3f6i7fyjJ44bx\nm4f3AQAAAMCSLRZgPT1JJ3lVd39gmgmGcf8zk+2HT5/mHQAAAADMrsUCrFOH9v0rnGf+7KtTD9oL\nAAAAABZYLMCa3yq4pK8VHsT8+JtX+B4AAAAAZsxiAdanh/YZK5znmQveBwAAAABLsliAtT2Ts6t+\noqqeP80Ew7ifyOQsrXdM8w4AAAAAZtdiAdark1yVyRbAN1XVe6vqe6rqpIMNqqqThn7vTfKmJBuS\n7B7eBwAAAABLdtCzrbp7T1U9M8n/SXK3JI8brlTVtUk+l+T6JDclOTrJcUnukeT4fV5TSb6Y5Jnd\nfe1q/wMAAAAAOLItejh7d+DQMP0AACAASURBVF9cVd+e5BVJvj+T1VRJsjm3Darm1T5/35zkj5L8\nfHdftcJaAQAAAJhBS/q6YHd/PslZVfXTmRzI/ogk98tktdUdk2xKMpfkq5msyro0yd8keWd3/+sh\nqBsAAACAGbGkAGted1+d5PeHCwAAAAAOucUOcQcAAACANSXAAgAAAGDUlrWFcF9VtSmTM7COy61n\nYF2f5HPdPbc65QEAAAAw65YcYFXV7TI5wP1ZmRzifmpu+8XBeV1VV2RyiPs7kvx5d9+yCrUCAAAA\nMIOWFGBV1eOT/FaS+8zfOlj3JKcluWeSFyTZVVU/1t3vX0GdAAAAAMyoRQOsqnphkt9LsiG3BleX\nJ7ksyZWZbBu8McntM9lOeEqS+yb590Pf+yZ5T1X9YHf/0apWDwAAAMAR76ABVlWdnuS1Q7+vJfmV\nJK/v7qsWe3FVnZzkB5L8dCbB1v+uqr/v7v83TaFVdWqSZyd5XJL/mORuSW5K8qkk70nym939+f2M\n25Lk+5J8R5JvS3LXJMcnuTbJJ5K8PcnruvvGaeoCAAAA4NCq7j7ww6rXJPmRJF9K8sju3rXsCSYh\n2N8kOSHJ73T3j03xjlOSfDa33bp4XZJjM1kZliTXJHlOd//VgrHPT/In+9y6cbjutM+9TyZ5Qnfv\nXko9W7du7YsuumhZ/wYAAAAADqyqLu7urft7drtFxj4xSSd5xTThVZIM4345k/Bp2zTvyK0h1QVJ\nnpvkhO4+PskdkjwlyaeT3DnJ9qo6acHYK5L8QpLHJrlLdx8zjD0+yUsy2QL5H5L84ZS1AQAAAHAI\nLXYG1pah/bsVzjM/fstBex3YNUnO6O6P7Xuzu2/K5HytpyTZmcmqqpdmEljN9/m77Kf+7r4uye9V\n1S1Jfj/JY6vqlO6+csoaAQAAADgEFluBNTe0x61wnvnxcwftdQDdfe3C8GrB88uSfHj4+aBlvv4f\n9vn75OXWBgAAAMChtViAdfnQft8K53nB0E51gPsSfXloNxy017/1sH3+/szqlAIAAADAalkswHpr\nJmdXnVVVPzvNBMO4szI5S+ut07xjCXMcleThw8+PL6H/0VV1r6r6ySS/Ntx+W3d/8VDUBwAAAMD0\nFjsD63eSvDjJfZP8clW9MMkbk/x1kkuHc6Ruo6rulOR+SR6d5EVJ7jM8+mSS165K1f/Wy5KclOSW\nJOcdqFNV/XOSb11wu5O8LckPHKLaAAAAAFiBgwZY3X1jVT01k6//3TeTMOoV88+r6muZfMXvpiRH\nZ3LW1bELXlOZhFdP6+4bV6/0b9bwwCSvHH6+prsvPUj3q/epcf5crrcm+fnuvn61awMAAABg5Rbb\nQpju/nSSB2fyZb89mQRS89dxmax8OnVoj1vwfE+S/y/Jd3T3Z1a7+Kq6e5LtSTYluTjJzyzyb3lo\nd5/U3Xcc6v25JE9LcklVPXeRuV5SVRdV1UVXX3316vwDAAAAAFhUdffSO0/OmvquJI/IZJvgPZLc\nMZMAaS7JV5N8LsmlSf4myYXd/Y3VLfmbtZyQyVbG+2dy2PwjpznDqqqenuSdSb6e5D7dvXuxMVu3\nbu2LLrpouVMBAAAAcABVdXF3b93fs8XOwLqNIYx633Ctmao6PsmOTMKrK5I8ftoD2Lv7XVX12ST3\nTPL83HqoOwAAAAAjsOgWwrGpqmOTvDvJ1iRfyCS8umKFr51fdbXwgHcAAAAA1ti6CrCqalOSdyV5\nWJIvZxJeXb4Krz5taB3kDgAAADAyy9pCOK+qNiT590lOyeTg9vkzsK5PcmWSy7v75tUqcpjz6CTn\nJ3lMJofDP7G7P7GEcUcd7ByuqnpBkpOHnx9cjVoBAAAAWD1LDrCq6o5JfijJs5I8ZJGx36iqv88k\ncHp9d1+3kiKHwOzNSZ6UyUHxT+7ujy5x+Aeq6p2ZHNS+az5Yq6pTk5yV5L8O/S5OcsFK6gQAAABg\n9S3pK4RV9aIk5yY5Yf7WEt49/+JrkvxUd583TYHD/I/K5IuDSXJDkmsP0v3K7n7wPmM/k8kB7Umy\nN8l1SY5Jcuw+Y/4hyTO6+wtLqcdXCAEAAABW14q+QlhV5yT5pfmfmWwT/FCSyzLZLnh9khuT3D6T\n7YSnJLlvkocOv09I8vqqOqm7f2XKf8O+Z3UdM1wHcsOC3y/KZOXWo4baTkxyS5LPJPlokrcledtq\nb3kEAAAAYHUcNMCqqgcn+cVMgqurkvxskrd2902LvXg4s+p5SV6ZZEuSX6qq93f3spcudfeFWdqq\nrwONvXCasQAAAACsvcW+QvgjSTZkslrp27v7j5cSXiVJd9/U3X+cZGuSzw5z/cgKagUAAABgBi0W\nYD06k7OsXtHd/zrNBN39xSSvyGQF1XdN8w4AAAAAZtdiAdbdh3bnCueZH3/SCt8DAAAAwIxZLMC6\nbmjvusJ5Thzar67wPQAAAADMmMUCrE8M7UtWOM9LF7wPAAAAAJZksQDrzZmcXfWMqnptVd1hOS+v\nqjtU1WuTPCOTs7T+eLoyAQAAAJhVRy3y/PVJzkry0ExWYT2vqt6a5K+TXJrkc0mu7+6bquroJMcl\nuUeS+2VyAPzzkmwe3vV3Sd6w6v8CAAAAAI5oBw2wuvuWqnpaknckeVSSO2cSZN1mS2FVHegV8w/+\nOsmzu7tXVC0AAAAAM2exLYTp7muSPCaTlVgfzySUWur18SQvSvLY4T0AAAAAsCyLbSFMkgwrp85L\ncl5V/bskj8hkm+A9ktwxyaYkc5l8ZfBzmWwv/Jvu/udDUTQAAAAAs2NJAda+hlBKMAUAAADAYbHo\nFkIAAAAAWEsCLAAAAABGTYAFAAAAwKgdlgCrqrZV1V9W1fsPx3wAAAAAHDmWfYj7lE5K8l1J+jDN\nBwAAAMARwhZCAAAAAEZNgAUAAADAqB10C2FVPWqV5rnvKr0HAAAAgBmz2BlYF8a5VQAAAACsoaUe\n4l6HtAoAAAAAOIDFAqybkmxM8k9J3rGCef5TkmeuYDwAAAAAM2qxAOufkmxNsre7f2HaSarqhRFg\nAQAAADCFxb5C+A9D+4CqOvpQFwMAAAAACy0WYH1kaDdmsg0QAAAAAA6rpQZYSfLgQ1kIAAAAAOzP\nYmdgXZbkRZl8hfDSaSfp7vOSnDfteAAAAABm10EDrO7uJH94mGoBAAAAgH9jsS2EAAAAALCmFttC\nCABwRNq+c3fO3bErV+2Zy8mbN+XsbafnzDO2rHVZAADshwALAJg523fuzjnnX5K5vTcnSXbvmcs5\n51+SJEIsAIARsoUQAJg55+7Y9c3wat7c3ptz7o5da1QRAAAHI8ACAGbOVXvmlnUfAIC1JcACAGbO\nyZs3Les+AABrS4AFAMycs7ednk0bN9zm3qaNG3L2ttPXqCIAAA7GIe4AwMyZP6jdVwgBANYHARYA\nMJPOPGOLwAoAYJ2whRAAAACAUVv2Cqyq+vXhzwu7+52rXA8AAAAA3MY0Wwh/Ymjfv5qFAAAAAMD+\nTLOF8MtDu3s1CwEAAACA/ZkmwLp8aE9ezUIAAAAAYH+mCbD+NEkl+Z5VrgUAAAAA/o1pAqzXJtmZ\n5AVV9UOrXA8AAAAA3MY0h7jfPcmLk7w+ye9W1fOTvCnJx5Jck+Tmgw3u7iummBMAAACAGTVNgPWZ\nJD38XUkeM1xL0VPOCQAAAMCMmjZMqgP8DQAAAACrapoA66xVrwIAAAAADmDZAVZ3n3coCgEAAACA\n/ZnmK4QAAAAAcNgIsAAAAAAYtRV9EbCqbpfJFwgfmuSkJHdI8nPd/fl9+hw9zHNzd9+4kvkAAAAA\nmD1TB1hV9ZQkv5XktAWPfjXJ5/f5/eIkr05yfVWd3N1fm3ZOAAAAAGbPVFsIq+qsJO9Mcq8kleQr\nQ7s/v5/kuiTHJTlzmvkAAAAAmF3LDrCq6t5J/ncmgdVfJ/m27j7xQP2HbYPnD/2fOGWdAAAAAMyo\naVZgvTzJxiSXJXlSd39yCWM+OLRnTDEfAAAAADNsmgDr8Uk6yf9axqHslw/tKVPMBwAAAMAMmybA\nmg+hPrqMMdcP7bFTzAcAAADADJsmwJo/rP2YZYy589BeN8V8AAAAAMywaQKszw/tvZcx5qFD+9kp\n5gMAAABghk0TYH0gk1VY37uUzlW1KclLMzk368Ip5gMAAABghk0TYL1+aLdV1dMP1rGq7pDkLUlO\nTXJLkt+bYj4AAAAAZtiyA6zu/tskf5TJKqw/q6pXV9Wj9ulyn6p6bFX99ySXJXlaJquvXtPdl61G\n0QAAAADMjqOmHPfiJJuTPD3Jy4arh2d/tk+/+QPf35bkp6acCwAAAIAZNs0WwnT3Td39zEyCrH/J\nJKja33Vlkh/u7u/p7ltWp2QAAAAAZsm0K7CSJN39B0n+oKrul2Rrkrsm2ZDkS0l2JtnZ3X2QVwAA\nAADAQa0owJrX3ZcmuXQ13gUAAAAA+1r2FsKqmmrbIQAAAABMY5owandV/UZVbV31agAAAABggWkC\nrLsl+fEkf19Vl1XVz1fVvVa5LgAAAABIMl2A9b4kt2TylcH7JPmFJP9cVX9bVT9cVSesZoEAAAAA\nzLZlB1jd/cQk90jyk/8/e/cerVtd14v//WEDskGuiYKkeanAG4Zuql9kSF7zrmTHvHsq7OTxUic1\nO9mJblp2s/R4KfFyOlqOUgov4anAy++YiRB5SaIUURREdAMKiMLn/PHMFYvtWmuv9axn7TUXz+s1\nxjO+z5zz+53fzx6DZ8B+853fmeQjmQRZleT7k7wyyReq6q+r6seq6lazLBYAAACA+TPVhuzdfVl3\n/0F3H5/kbkl+PclFmQRZ+yR5eJK3JLmsqk6rqgfMqF4AAAAA5sy63yjY3Rd09y93912TnJDkVUm+\nnEmYdVCSpyV5T1V9rqp+e73zAQAAADBf1h1gLdbdH+zuZyU5Msmjk7w1yXWZhFm3T/LfZjkfAAAA\nALd8Mw2wFnT3N7v7jCRPSfKsJDs3Yh4AAAAAbvn23oibVtWJSZ6c5OQkB2/EHAAAAADMh5kFWFV1\nbJInJfnxJEctnB7aKzJ5nPBPZzUfAAAAAPNhXQFWVd0hyRMzCa7usXB6aK9LckYmodW7u/ub65kL\nAAAAgPm05gCrqg5N8vhMQqsTMgmsFkKrG5OcnUlo9RfdffVsygQAAABgXk2zAusLSfYZvi8EVx/N\nJLR6c3dfMovCAAAAACCZLsDad2g/l+QtSf60uz86u5IAAAAA4CbTBFivz2S11dnd3TOuBwAAAABu\nZs0BVnf/xEYUAgAAAABL2WuzCwAAAACAlUzzCOF/qKp9kvxokgcluVeSw4ZLX85kY/f/k8nbCL+x\nnnkAAAAAmF9TB1hV9SNJ/jjJkYtPD+2dktwnydOS/HZV/VR3/820cwEAAAAwv6YKsKrqSUnemElg\ntRBaXZTk0uH7EZmEWElyVJJ3VNVTuvstU1cK3Mzp512Sl515QT6/89rc/pDtef5Djs5jjjtqs8sC\nAACAmVvzHlhVdYdMVl7tleTaJL+c5Mjuvkt3/8DwuUsmIdaLk3xt6Psnw1hgnU4/75K86G0fzSU7\nr00nuWTntXnR2z6a08+7ZLNLAwAAgJmbZhP35yXZL8k1Se7f3b/e3Zft2qm7v9jdv5HkxKHvfkme\ns55igYmXnXlBrv3GDTc7d+03bsjLzrxgkyoCAACAjTNNgPXgJJ3kd7v7nN117u5zk/xeJo8aPmSK\n+YBdfH7ntWs6DwAAAFvZNAHWHYf2PWsYc+bQfscU8wG7uP0h29d0HgAAALayaQKshY3fv76GMQt9\np37rIXCT5z/k6GzfZ9vNzm3fZ1ue/5CjN6kiAAAA2DjTBFgLbxrcsYYxC30vXbEXsCqPOe6ovORx\n98pRh2xPJTnqkO15yePu5S2EAAAA3CJNsyLqfUnunORFVfXn3f2VlTpX1SFJXpjJvlnvm2I+YAmP\nOe4ogRUAAABzYZoVWK8e2m9P8sGq+qHlOlbVDyb5QG7a++pVU8wHAAAAwBxb8wqs7v5QVb08yXOT\nfFeSs6rqU0n+IckXM1lpdbsk35vkOxcNfXl3/+O0hVbVHZM8LskDktx7mOP6JJ9K8u7h/l9YYtxB\nSR6dydsTj89kE/pK8vlMVoT9YXefN21dAAAAAGysqTZV7+6fraprk7wgk1Vcd01yl1261dDemOSl\n3f3fpy2yqu6Q5KJF90ySq5IckOTY4XNKVZ3c3WftMvwjuXmQds3Q3mX4PKWqXtjdvzttfQAAAABs\nnGkeIUySdPcvZhIc/c8kF2YSLi3+XDhcO3Y94dVg4XVr70zy+CSHdffBSfZP8rAkn05yaJLTq+qI\nXcbuk+S8JM9KcqfuPiDJrTNZxXXWcO/fqaqHrbNGAAAAADZAdfdsblS1byYhUpJ8pbuvn8mNJ/c+\nOJPw6fxlrh+TSUi1X5Jf6e5TF127X3e/f5lx2zNZoXW3JGd390mrqWfHjh19zjnnrPFPAQAAAMBy\nquoj3b1jqWtTr8DaVXdf392XDZ+ZhVfDva9cLrwarn8ykz24kuS+u1xbMrwarl2b5M+XGgcAAADA\nOMwswBqBK4Z224q9ZjcOAAAAgD3gFhFgVdXeSU4YDj+2xuEnTjkOAAAAgD3gFhFgZbJB+xGZvPHw\njasdVFX3SfLY4fD1G1AXAAAAAOu05QOsqjo2yUuGw1d09ydWOe7AJP87k0cHz03yJ7vpf0pVnVNV\n51x++eXrKRkAAACANdjSAVZVHZnk9CQLbxN84SrH7Z3kzUmOSbIzyRO6+5srjenu13b3ju7ecfjh\nh6+vcAAAAABWbcsGWFV1WJL3JLlzkguTPLy7r1vFuL2SvCHJI5Jck+SR3X3hBpYKAAAAwDpsyQCr\nqg5OcmaSeya5OMkDu/uyVYyrJK9K8qQk1yd5bHd/YCNrBQAAAGB9tlyAVVUHJHlXkh1JLs0kvLp4\nlcP/IMkpSb6ZyWOD79mYKgEAAACYlS0VYFXV9iRnJPmBJFdkEl6t6vG/qnppkudk8qbCp3X32zes\nUAAAAABmZssEWFW1b5K3JTkpk43XH9zdH1/l2BdnssF7Jzmlu9+8YYUCAAAAMFNbIsCqqm2ZvDXw\noUmuTvIj3X3uKsc+L8mvDofP7u7XbUyVAAAAAGyEvZe7UFU3bMB83d3LzrmCE5KcPHzfJ8npk/3Y\nl/TZ7j5+0fHvDe2NSV48rMZazvHd/dkp6gMAAABgg6wUJi2bEG2CxSvF9hs+y7lul+OFP8deSW63\nm3m2rbEuAAAAADbYSgHWqbsZ+/BM3gSYJB9P8o9JLhuOb5fk+CT3zGTfqXMyeXPgVLr77EwZqHX3\nmII4AAAAANZo2QCru5cNsKrqlzMJr87PZFP0Dy/T7/gkrxn6vrO7f3WpfgAAAACwnDVv4l5VD0jy\nK0kuSPKDy4VXSTJcu1+SC5P8j6p64JR1AgAAADCnpnkL4XMyeSzwpd39td11Hvq8NJNHAJ89xXwA\nAAAAzLFpAqyFfa8+uoYx/zy0x6/YCwAAAAB2MU2AddjQHrKGMQcP7aFTzAcAAADAHJsmwPr80P7o\nGsY8fmi/MMV8AAAAAMyxaQKsv8lkP6ufqqqn7K5zVT0pySmZ7Jv1rinmAwAAAGCOTRNg/WaSq4ax\nb6iqv6mqJ1TVXavqwKq69fD9CVX17iRvGvpeneQlsysdAAAAgHmw91oHdPclVfXIJGckOSjJg4bP\nciqT8OrR3X3JVFUCAAAAMLemWYGV7n5/knsl+cskN2YSUi31uTHJ25Ic293vnUXBAAAAAMyXNa/A\nWtDdn03y+Kq6XZKTMgm0Ft5Q+JUkH01yVndfuu4qAQAAAJhbUwdYC7r7siR/NnwAAAAAYKbW/Ahh\nVZ07fJ65EQUBAAAAwGLTrMC6VybB1ydnXAsAAAAAfItpNnG/bGivnmUhAAAAALCUaQKs84b26FkW\nAgAAAABLmSbAel2SSvJfZlwLAAAAAHyLNQdY3X16kjcl+cGqelNV3Xr2ZQEAAADAxJo3ca+qpyY5\nK8n3JHlSkkdU1RlJzk/ylSQ3rDS+u980RZ0AAAAAzKlp3kL4hiS96PiQJE8ePrvTmazeAgAAAIBV\nmSbASiZ7YK10DAAAAAAzMU2AdeeZVwEAAAAAy1hzgNXdn9mIQgAAAABgKWt+CyEAAAAA7EkCLAAA\nAABGTYAFAAAAwKhN+xbCJElVnZTkMUnuneQ2SbZn5TcSdnffdT1zAgAAADBfpgqwquq2Sf4syYkL\np5bp2rtc62nmAwAAAGB+rTnAqqp9krw7yfdkEk79U5JLkjw8k4DqT5McluQ+SY4czp2b5GOzKRkA\nAACAeTLNHlhPT3Lc8P0Z3X2fJL+wcLG7n9bdj+zuo5I8LskXktw9yTu6+xnrrBcAAACAOTNNgHXy\n0P5Nd79xpY7dfXomjxlen+QNVfVdU8wHAAAAwBybJsC6d256VPBbVNXN9sPq7n9P8vIkByR57hTz\nAQAAADDHpgmwDhvaTy86d/2i7/svMebvhvZBU8wHAAAAwBybJsC6fpc2Sa5a9P2oJcZct8I1AAAA\nAFjWNAHWxUN7u4UT3X1ZkquHw+9bYsw9F7pOMR8AAAAAc2yaAOvcoT1ul/PvS1JJnltVt1o4WVWH\nJHlhJuHVJ6YpEgAAAID5NU2A9XeZBFUP3+X8q4f2uCT/XFUvq6r/meSjSb57uPamqaoEAAAAYG5N\nE2CdnsljhN9eVXddONnd70xyWibh1ncl+bkkz8xN+169J8mr1lUtAAAAAHNn77UO6O6dSe60zLWf\nrKoPJvnJJPcY7n9hJiuvXt7dN05fKgAAAADzaM0B1u509+uSvG7W9wUAAABgPk3zCCEAAAAA7DFr\nDrCq6rYbUQgAAAAALGWaRwi/UFWfTPLeJGcnObu7vzjTqgAAAABgME2AVUnuluSYTN4ymKq6IEOY\nFYEWAAAAADM0TYD17CQnJfmhJLcZzh2T5OgItAAAAACYsTUHWN39yiSvTJKqumeS+w+fVQVa3f3W\nddYMAAAAwByp7p7dzZYPtBbc2N3TrPoalR07dvQ555yz2WUAAAAA3GJU1Ue6e8dS19b8FsKVdPfH\nuvsV3f2jSe6R5FeTXJlkISWrWc4HAAAAwC3fzFZDVdWhSU7MZH+s+2cSYFVuHlp9ZlbzAQAAADAf\npg6wquqQTAKr+2cSWt0zN4VVC+3FSd6b5KxM9r+6aNr5AAAAAJhPaw6wqur3Mgmtjs23rrD6XG56\n++BZ3f3pdVcIAAAAwFybZgXW8zLZ06qSXJKbr7D69xnWBgAAAADr2sS9k3whyeeHz2UzqQgAAAAA\nFpkmwHprki9msgJrR5L/luQdSb5cVR+qqpdW1UOr6oAZ1gkAAADAnFrzI4Td/YQkqapjMtkL6/6Z\nbOZ+uyTHZxJqPT/JN6vq3Az7YSX5QHdfM4uiAQAAAJgfU7+FsLs/meSTSV6dLBtofV+S703ygkwC\nrXO6+4T1lQwAAADAPJk6wNrVMoHWE5I8N8nBSfZJ8v2zmg8AAACA+TCzACtJqurbMlmBddLwOWa4\ntPDWQgAAAABYk3UFWFV1aCaPCy4EVvdYfHnR968keX8me2EBAAAAwKqtOcCqqkflplVW98pNQdXi\nwGpnbgqszk5yfnf3egoFAAAAYD5NswLr9HzrI4FX5uaB1T8JrAAAAACYhWkfIbwqyQdyU2B1nsAK\nAAAAgI0wTYB1fCaB1Y2zLgYAAAAAdrXmAKu7P7IRhQAAAADAUvba7AIAAAAAYCXrCrCq6qSq+l9V\n9W9V9dWq+mZV3X2XPj9UVT9TVU9eX6kAAAAAzKOpNnGvqu1JXp/k8QunhnapjdxvTPKKJF1VH+ru\nC6eZEwAAAID5NO0KrLdkEl5VknOS/O5yHbv7A0k+MRw+bsr5AAAAAJhTaw6wqurRSR41HP5Md39f\ndz9/N8PenknYdeJa5wMAAABgvk2zAuvpQ/tn3f3qVY758NDebYr5AAAAAJhj0wRY35vJXldvXsOY\nLwztbaeYDwAAAIA5Nk2AdZuhvWQNY25Yx3wAAAAAzLFpAqWrh/aINYy549BeMcV8AAAAAMyxaQKs\nfxvae6xhzCOG9mNTzAcAAADAHJsmwPqbTN4o+Oyq2nt3navquCRPzmTfrHdOMR8AAAAAc2yaAOsV\nmTxGeIckr6+qWy3XsaoenUngtW+SLyV53TRFAgAAADC/druCalfd/aWq+qkkb0nyxCQPqqp3Lery\ngqraK8kJSe6UyWqtG5I8pbuvWX/JAAAAAMyTNQdYSdLdb62qG5L8SZLbJnlaJo8IJslThraG9qok\nT+3u96ynUAAAAADm0zSPECZJuvsvk9w1yS8n+Ugmq6wqNwVXH0vyG0m+s7v/ep11AgAAADCnplqB\ntaC7v5zk15P8+vDY4GFJtiW5oru/OYP6AAAAAJhzaw6wqupRw9eLuvufF853942ZbNQOAAAAADMz\nzSOEpyd5e5J7zbgWAAAAAPgW0wRYVw3tv8yyEAAAAABYyjQB1meH9tazLAQAAAAAljJNgHXG0D5o\nloUAAAAAwFKmCbBenuSKJM+tqnvPuB4AAAAAuJk1B1jdfVmSRyS5MskHquqXqurOM68MAAAAAJLs\nvdYBVfWp4eutkhyQ5NQkp1bVV5PsTHLDCsO7u++65ioBAAAAmFtrDrCS3GmX4xraA4fPSnqK+QAA\nAACYY9MEWG+ceRUAAAAAsIw1B1jd/YyNKAQAAAAAljLNWwgBAAAAYI8RYAEAAAAwagIsAAAAAEZN\ngAUAAADAqAmwAAAAABi1LRNgVdUdq+p5VXVGVV1cVV+vqqur6vyqemlVHbnMuL2q6qSqekFVvbWq\nPl1VPXx+ek//OQAAAABYm703u4DVqKo7JLkoSS06fVWSA5IcO3xOqaqTu/usXYYflOTv90SdAAAA\nAMzeVlmBtW1o35nk8UkO6+6Dk+yf5GFJPp3k0CSnV9URS4z/WpL3J/n9JE9McumGVwwAAADATGyJ\nFVhJvpLkuO4+f/HJfJpWJAAAIABJREFU7r4+ybur6mFJzstktdUzk5y6qNuVSQ7q7hsXTlTVSza+\nZAAAAABmYUuswOruK3cNr3a5/skk/zAc3neXa704vAIAAABga9kSAdYqXTG021bsBQAAAMCWcosI\nsKpq7yQnDIcf28xaAAAAAJitde2BVVX3SnJikrskOTC7X/3U3f0T65lzGc9KckSSG5O8cQPuDwAA\nAMAmmSrAqqq7JjktyQ+uZViSTjLTAKuqjk2ysCn7K7r7E7O8/6J5TklySpLc8Y533IgpAAAAAFjC\nmgOsqrpdkvdlsuKphtNfzeRNgXt0s/SqOjLJ6Um2J/lIkhdu1Fzd/dokr02SHTt29EbNAwAAAMDN\nTbMC6xeTHJnJaqrXJ/nt7r5gplWtQlUdluQ9Se6c5MIkD+/u6/Z0HQAAAABsrGkCrIdnEl69eYP2\ns9qtqjo4yZlJ7pnk4iQP7O7LNqMWAAAAADbWNG8hPGpo3zDDOlatqg5I8q4kO5Jcmkl4dfFm1AIA\nAADAxpsmwLpqaK+YZSGrUVXbk5yR5AeG+R/Y3Rfu6ToAAAAA2HOmCbD+eWjvPMtCdqeq9k3ytiQn\nJdmZ5MHd/fE9WQMAAAAAe940AdarM3n74NNnW8ryqmpbkjcneWiSq5P8SHefu4bxB1fVbRY+uenP\nfcDi81V1q9lXDwAAAMB6rDnA6u6/TPKnSR5RVS+efUlLOiHJycP3fZKcXlWXLvP58BLj/yrJ5Ys+\ndxjO/84u5398Q/8UAAAAAKzZmt9CWFU/lOS0JHdJ8itV9egk/zvJvyS5Znfju/t9a50zNw/a9hs+\ny7luivsDAAAAMFJrDrCSnJ2kFx0fN3xWo6eZs7vPzuSxxal09/2nHQsAAADA5pomwErWESYBAAAA\nwFpME2CdNPMqAAAAAGAZ0zzO996NKAQAAAAAlrLmtxACAAAAwJ4kwAIAAABg1ARYAAAAAIzatG8h\nTJJU1V2TPCrJvZPcJsn2rPyGwu7uB6xnTgAAAADmy1QBVlXtn+SVSZ6Sbw2sKkkvcS5LnAcAAACA\nFa05wKqqSvL2JA/MJJj6UpLPJfmeTAKq9yc5LMnRw/07yQVJLp1NyQAAAADMk2n2wHp8kgcN309N\nckSSpy5c7O4Tu/teSQ5N8nNJvpZJoPXi7j5pfeUCAAAAMG+mCbCeOLQf7O5Tu/vGLPFoYHd/rbv/\nIMkDkhyY5G1VdfvpSwUAAABgHk0TYO3IJLD649V07u4PJ3lVJpu8P2eK+QAAAACYY9MEWLcZ2k8t\nOveNhS9VtX2JMe8c2kdMMR8AAAAAc2yaAOubQ3v1onOLvx+xxJgrh/YOU8wHAAAAwBybJsD6/NAe\nvujcpUmuHb7fZ4kx3zW0a37rIQAAAADzbZoA6/yhvdfCie7uJB8aDn9mceeq2ieTtxEmyYVTzAcA\nAADAHJsmwPr7JJXkobucP204f/+qOruqnlVVL0jyj7lp4/e3rqdYAAAAAOZPTRZPrWFA1RFJLkly\nY5Kju/tTi669K5Nga9ebVpLzkpzQ3detq+IR2LFjR59zzjmbXQYAAADALUZVfaS7dyx1bc0rsLr7\n0iT7JNlvcXg1eGyS30hyWSahVWWygfsrk5x0SwivAAAAANizptpUvbtvXOb815O8OMmLq+qw4f6X\n91qXeQEAAADAYMPeCtjdX96oewMAAAAwP6bZxB0AAAAA9ph1rcCqqoOSPD7J/5fkiCT7J3lGd39m\nUZ/bJzkkyXVL7JkFAAAAACuaOsCqqv+S5CVJDlw4lcnbBw/YpetJSd6U5OtV9e0eLQQAAABgLaZ6\nhLCqfinJK5IclOT6JOeu0P0tSb6Y5FZJTp5mPgAAAADm15oDrKq6d5JTh8O3JDmyu49frv/wxsK/\nzGSF1gOnKRIAAACA+TXNCqxnZxJGfSjJk7t75yrG/N+hPXaK+QAAAACYY9MEWCdmstfVK7u7Vznm\n00N7+ynmAwAAAGCOTRNgLYRQn1jDmGuGdr8p5gMAAABgjk0TYH1zaA9aw5jDh/bKKeYDAAAAYI5N\nE2B9bmi/cw1jThzaf59iPgAAAADm2DQB1tmZbOL+n1fTuaq+LckzM9k362+nmA8AAACAOTZNgPWq\nJDcm+f6q+pmVOlbVkUnekeQ2Sa5P8pop5gMAAABgjq05wOrujyX5nUxWYf1RVf11VT11UZeTquo/\nV9VpSf41yfdmsvrqV7r7c996RwAAAABY3t7TDOruX6iq/ZP81yQPHz49XP7DRV1raH+nu39r6ioB\nAAAAmFvTPEKYJOnu5yR5cJKzMnmksHb5JMn/n+Sh3f2CddYJAAAAwJyaagXWgu7+2yR/W1UHJjku\nyW2TbEvypSTnd/eX1l8iAAAAAPNsXQHWgu6+Osn7ZnEvAAAAAFhs6kcIAQAAAGBPEGABAAAAMGor\nPkJYVafMesLufu2s7wkAAADALdfu9sB6dZKe4XydRIAFAAAAwKqtdhP32tAqAAAAAGAZqw2wrkny\nV0nemmTnxpUDAAAAADe3uwDr35J8Z5L9kzwhyWOTvCPJm5K8u7tv2NjyAAAAAJh3K76FsLu/O8kJ\nmexbtTPJfklOzmQ11iVV9ftVdZ8NrxIAAACAubVigJUk3f3B7v7pJEcm+bEk70pyQ5LbJnlOkg9X\n1ceq6gVVddSGVgsAAADA3NltgLWgu6/v7r/o7kcmOSrJzyX5p0w2eL97kpckuaiq3lNVT66q/Tek\nYgAAAADmyqoDrMW6+/Lu/oPuvm+Seyb5nSRfSLItyQOTvDHJpVX1JzOrFAAAAIC5NFWAtVh3f6K7\nX5DkjkkemuTNSa5PcuskT1zv/QEAAACYb+sOsBY5MMl3DJ99Z3hfAAAAAObY3usZXFXbkjwsyVOS\nPCLJrTLZEytJ/jXJG9ZzfwAAAACYKsCqquMzCa2ekOTbclNo9eUkf57kTd39oZlUCAAAAMBcW3WA\nVVV3yCS0ekqS7144neQbSd6V5E1J3tHd35h1kQAAAADMrxUDrKo6MMmPJnlqkvtlElgtrLb6x0xC\nqz/r7i9vZJEAAAAAzK/drcC6LDff1+riJH+aySOC/7qRhQEAAABAsvsAa78kneTaJH+V5L3D8f2r\n6v7TTNjdr51mHAAAAADzabV7YO2X5D8Nn/XoJAIsAAAAAFZtNQFW7b4LAAAAAGyM3QVYJ+2RKgAA\nAABgGSsGWN393j1VCAAAAAAsZa/NLgAAAAAAViLAAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAA\nMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNg\nAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAA\ngFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoA\nCwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAA\nAIzalgmwquqOVfW8qjqjqi6uqq9X1dVVdX5VvbSqjtzN+H2r6gVV9U9V9dWq2llVH6yqU6qq9tSf\nAwAAAIC12XuzC1iNqrpDkouSLA6arkpyQJJjh88pVXVyd5+1xPiDkvx9kvsOp65Jsj3J9w+fR1bV\nY7v7mxv2hwAAAABgKltlBda2oX1nkscnOay7D06yf5KHJfl0kkOTnF5VRywx/o8zCa++nOSRSW49\njH16kuuSPCLJqRtYPwAAAABT2ioB1leSHNfdj+juv+juryRJd1/f3e/OJMS6LslBSZ65eGBVHZfk\nx4bDZ3T3O3rihu5+Y5JfGK79bFXddo/8aQAAAABYtS0RYHX3ld19/grXP5nkH4bD++5y+YlDe0F3\n//USw1+b5MpMHil83HprBQAAAGC2tkSAtUpXDO22Xc6fNLTvWWpQd1+b5P3D4Q9vQF0AAAAArMMt\nIsCqqr2TnDAcfmzR+UpyzHD48RVu8YmhvfvsqwMAAABgPW4RAVaSZyU5IsmNSd646PxBmbypMEk+\nv8L4hWtHzr40AAAAANZjywdYVXVskpcMh6/o7k8sunzAou/XrnCba4b21ivMc0pVnVNV51x++eXT\nFQsAAADAmm3pAKuqjkxyeiYbsH8kyQs3aq7ufm137+juHYcffvhGTQMAAADALrZsgFVVh2WyMfud\nk1yY5OHdfd0u3b626Pv2FW63/9B+dXYVAgAAADALWzLAqqqDk5yZ5J5JLk7ywO6+bImuV+WmEOv2\nK9xy4doXZlYkAAAAADOx5QKsqjogybuS7EhyaSbh1cVL9e3uTvIvw+E9VrjtwtsHP7FCHwAAAAA2\nwZYKsKpqe5IzkvxAkisyCa8u3M2ws4b2Qcvcc78k9xsO/24WdQIAAAAwO1smwKqqfZO8LclJSXYm\neXB3f3wVQ98ytMdU1SOWuP5TSQ7O5C2Fb59FrQAAAADMzpYIsKpqW5I3J3lokquT/Eh3n7uasd19\nXpK3DodvqKqHLdyzqp6a5LeGa7/f3V+cbeUAAAAArNfem13AKp2Q5OTh+z5JTq+q5fp+truP3+Xc\nTyW5a5L7JnlnVV2TZFuSWw3X35Hkf8y0YgAAAABmYqsEWItXiu03fJZz3a4nuvuqqvqBJD+b5MeT\nfGeSryc5L8nrk/zxsOE7AAAAACOzJQKs7j47ybJLrlZ5j+szeVzwt3bXFwAAAIDx2BJ7YAEAAAAw\nvwRYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACjJsAC\nAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAA\noybAAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEW\nAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAA\nGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmw\nAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAA\nwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2A\nBQAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFETYAEAAAAwagIsAAAA\nAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAAAIBRE2ABAAAAMGoC\nLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACAURNgAQAA\nADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgALAAAAgFET\nYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAAjJoACwAA\nAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYtS0TYFXVgVX1qKr6tap6d1V9qap6+Byz\nivF3r6rTquqiqvr6MP5vq+rH9kT9AAAAAExn780uYA0ekOTt0wysqiclOS3JvsOpnUkOGu75gKp6\nRJKndXfPolAAAAAAZmfLrMAafDHJu5KcmuSU1QyoqvsmeX0m4dUZSe7c3YcmOTDJTye5PslTkvzC\nRhQMAAAAwPpspRVYZ3T36QsHVXWnVY77pST7JLkoyeO7++tJMrSvqarbZRKI/WJVvaa7vzzLogEA\nAABYny2zAqu7b1jrmKraluTBw+GrFsKrXfx+kk5y6ySPnb5CAAAAADbClgmwpnSbJPsP3y9YqkN3\nX53k88Phg/ZEUQAAAACs3i09wFq8Kfu2FfotPEp5jw2sBQAAAIAp3NIDrCuSfG34fvelOlTVYUlu\nNxweuSeKAgAAAGD1btEB1rBv1t8Nhz9TVQcs0e2Fi74fuPFVAQAAALAWt+gAa/CbSW7IZHXVu6vq\ne6tq36o6oqpenOTnk3xj6HvjcjepqlOq6pyqOufyyy/f+KoBAAAASDIHAVZ3fyjJKUm+meR+ST6U\n5OtJvpDkV5P8U5LThu47V7jPa7t7R3fvOPzwwze2aAAAAAD+wy0+wEqS7j4tyfckeU2Sjyb5bCZB\n1guTnJBkv6HrhZtSIAAAAADL2nv3XW4ZuvvjSX56qWtVdZ/h6wf3XEUAAAAArMZcrMBaSVXdI8m9\nhsM3b2YtAAAAAHyruQ6wqmrfJK8cDt/d3edvZj0AAAAAfKstFWBV1W0WPkkOXXTpkMXXqmqvXca9\noqruV1UHDMd7VdX9kvx9khOTXJ5lHi8EAAAAYHNttT2wLl/m/K57V905yUWLjp81fFJVO5MckGSf\n4dpFSR7Z3RfPrEoAAAAAZmZLrcBahxcmOTPJ55Lsn+TqTEKvn09y9+7+2CbWBgAAAMAKttQKrO6u\nKcf9dpLfnnE5AAAAAOwB87ICCwAAAIAtSoAFAAAAwKgJsAAAAAAYtS21Bxazc/p5l+RlZ16Qz++8\nNrc/ZHue/5Cj85jjjtrssgAAAAC+hQBrDp1+3iV50ds+mmu/cUOS5JKd1+ZFb/tokgixAAAAgNHx\nCOEcetmZF/xHeLXg2m/ckJedecEmVQQAAACwPAHWHPr8zmvXdB4AAABgMwmw5tDtD9m+pvMAAAAA\nm0mANYee/5Cjs32fbTc7t32fbXn+Q47epIoAAAAAlmcT9zm0sFG7txACAAAAW4EAa0495rijBFYA\nAADAluARQgAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARYAAAAAIyaAAsAAACA\nURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGDUBFgAAAACjJsACAAAAYNQEWAAAAACMmgAL\nAAAAgFETYAEAAAAwagIsAAAAAEZNgAUAAADAqAmwAAAAABg1ARYAAAAAoybAAgAAAGDUBFgAAAAA\njJoACwAAAIBRE2ABAAAAMGoCLAAAAABGTYAFAAAAwKgJsAAAAAAYNQEWAAAAAKMmwAIAAABg1ARY\nAAAAAIyaAAsAAACAURNgAQAAADBqAiwAAAAARk2ABQAAAMCoCbAAAAAAGLXq7s2uYcupqsuTfGaz\n65iR2yT50mYXAVuA3wqsjt8K7J7fCayO3wqszi3pt/Id3X34UhcEWHOuqs7p7h2bXQeMnd8KrI7f\nCuye3wmsjt8KrM68/FY8QggAAADAqAmwAAAAABg1ARav3ewCYIvwW4HV8VuB3fM7gdXxW4HVmYvf\nij2wAAAAABg1K7AAAAAAGDUBFgAAAACjJsCaQ1V1RFW9vKr+vaquq6rLquqMqnrAZtcGY1BVd6yq\n5w2/i4ur6utVdXVVnV9VL62qIze7Rhijqrp1VX22qnr4PH2za4Ixqaqjq+qPquqCqvpaVV1ZVf9S\nVadV1YmbXR9stqraq6qeUVV/W1WXV9U3qmpnVX2oqv57VR242TXCRquqA6vqUVX1a1X17qr60qL/\ntjpmFeP3qqpTquqDw+/n6qo6r6qeX1X77ok/w0axB9acqapjk/x9km8bTl2V5NaZhJmd5Be7+6Wb\nVB5suqq6Q5LPJKlFp69KckCSbcPxV5Kc3N1n7eHyYNSq6g+SPHfRqWd09xs2qRwYlap6TpKXJVn4\ny8NXk+ydZL/h+HXd/ZObURuMQVXtn+SMJD+86PSVSQ7KTf9d9pkkP9zdn9rD5cEeU1WPSfL2ZS7f\nrbs/ucLYfZKcnuRhw6nrk9yQZPtw/OFMfkNfnVG5e5QVWHOkqrYn+etMwqvzktyzuw9OcmiS383k\nXwy/WVUP3rwqYdMthFTvTPL4JIcNv5P9M/kXwacz+c2cXlVHbE6JMD5VdZ8k/zXJhza7Fhibqnpm\nkpdnElj9VpLv6O4Du3t7kiOTPDXJ/93EEmEMXpxJeNVJXpTkkO4+JJOQ98eT7EzyHUn+ZNMqhD3n\ni0neleTUJKesYdyvZ/J3luuSPD2Tv8MckOSRSb6c5Pgkr5lloXuSFVhzpKqel+T3M/k/fsd09yW7\nXH97ksckObe777sJJcKmq6qDk9ypu89f5voxmQTA+yX5le4+dU/WB2NUVXtlElwdl8l/GJ07XLIC\ni7lXVXdK8vFM/hJxSnf/8aYWBCNVVZ9Jcsckp3X3Tyxx/elJXj8cHtbdX9mD5cEeU1XbuvuGRcd3\nyuR/oicrrMAa/uf6RUluleS53f2Hu1x/dCarszrJ93T3P8+8+A1mBdZ8edLQvnnX8GrwsqG9T1Ud\nvYdqglHp7iuXC6+G659M8g/DoaAXJp6dZEeSV3X3eZtdDIzMczMJrz4kvIIV3W5ol/v3yEcWfd9/\ng2uBTbM4vFqjkzMJr65M8tol7vtXSf41kyevnjh1gZtIgDUnhg0PF/6yfeYy3f4hk3/Yk8SG7rC8\nK4Z224q9YA5U1VFJfi3JZUl+aZPLgTFa+EvCWza1Chi/i4b2uGWuL/xd5rJl/mc8zLuThvZ93X3d\nMn3eM7Q/vMz1URNgzY+75abNDz++VIfuvjHJBcPh3fdEUbDVVNXeSU4YDj+2mbXASPxRkgOT/Hx3\nX7m7zjBPququSW47HJ5XVd8/vOH2iqq6tqo+WVUvq6rbrnQfmBMLKxSfUVW/MGzrkKrat6r+UyZb\noXSSn9+sAmHkFv4Ov+Tf9wefGNq7VVWt0G+UBFjz48hF3z+/Qr+Fa0eu0Afm2bOSHJHkxiRv3ORa\nYFNV/b/27jxMjqrc4/j3R8K+g2GTJaABQRBlCbvIksgqQUBlEUEQ8IoQAfGCig8CF0U2QVwvi0Fc\nrhAQrkBAFlGWsIQdZHkgrAkokhBCEkh47x/n9J3KTHf1pGd6emb693meeqqqz+mqUz01ydTb57xH\newJ7A7dHxG9a3R6zfmhEYftTwN+BPYCFSQ/i65Eexh+S9NE+b51Z/3I+cBHpS/czgWmSpgGzgN8D\n/wA+4/9vzGqqPMN353l/qbwMKA5gtY8lC9uzSuq9k9cD7mY2azZJHyP9QQXwk4h4oqy+2WAmaUng\nJ8B7pMCumXW1XGH7e6TcI1tGxDKkv7V2I800tSpwVe7la9aWct6fscDxwNz88rJ0PLMuDQxrQdPM\nBorKM393nvdhAD7zO4BlZtYNklYlzdqxOCmJ6Lda2yKzlvs+abao8xzMNaup+Ld2AHtHxERIqRsi\n4gbgy7l8PeCzfdw+s34jz6B2J3AOcAWwMekBewRwErAOcImkM2sexMwGNQew2sfMwvbiJfUqM3q8\n3cS2mA0oklYgJTxcG3gG2L0kMaLZoCfp46SZ1V4iBbLMrLri31M3RsRTnStExJ9JPbPAk+hYexsH\njAQujohDIuKRiJgZEc9GxA+AI3O9Ez3k1qyqyjN/d573YQA+8zuA1T6K42BXK6lXKZvSxLaYDRg5\ngegEYEPgRWDniHitta0ya7kfk2bh/DYgSUsVl0K9RfNrnu7c2lXx768uwasqZWs0sS1m/ZakDYBR\nefe8anUi4nLSTNALAXv2UdPMBpLK/znded5/OyJmNLk9vc4BrPbxD1LXdYCq31hIWojUfR06Zicw\na1s5x8/1wGbAVFLw6sXWtsqsX1grr8cBM6osFT/P+/4/xdrVE6RJP7or6lcxG5TWL2w/X1Lvubwe\n3rymmA1Ylb+3ynooVmYqfLLJbWkKB7DaRI6u3p93R9WotgUpUSLALU1vlFk/Jmlx4Dpga9K3fTtH\nxDOtbZWZmQ0kEfEOcHfeXa+kaqVsclMbZNZ/FQO9a5bUq3yBMuB6jpj1gdvyejtJi9WoU4kFDMjn\nfQew2stv8/rAnJC6sxPy+oFqORrM2oWkRYDxwA7ANGB0RDze2laZ9R8RMTwiVGspVD00vza8VW01\n6wfG5fUukroEsSTtDqybd6/vs1aZ9S8PF7a/Uq2CpD2BlfLuxKa3yGzgGQ/MIc2Ae3jnwvw7tB6p\nt+/v+rZpvcMBrPbyC+AF0hS0/5vHmiNpaUln0THzzcktap9Zy0kaQgr27kL6dm/XiJjU2laZmdkA\ndglpWMcQYLykkZBSN0jaBbg417sHB7CsTUXEc6QJcwDGSjpT0koAOZfiIcBluXwycG1ft9GsL0n6\nQGUBli8ULVcsy2mAAIiIqaQ8pQBnSfpifrZB0m7ApbnsdxHxSF9cR29ThIfatxNJG5O6C66YX3qL\nND3tQqRI7Ml5lg+ztiTpk8Bf8+5sYHpJ9ZciYvPmt8psYJFU+ePi0Ii4rJVtMesPJK0D3E5HkvYZ\npIBWZYKDJ0i9fV/p+9aZ9Q95hMgtzJ8Pawbpy/eK10hfLj7Yl20z62uFv6XqWTsiJhfetzBwDbBb\nfmkOMI+O/2/uA3YaiAncwT2w2k5EPEyaTe0CUhLERUn5ff4MjHLwymy+fxcXA1YuWYb1eevMzGzA\nyb1LNgLOIAWrhpK+OJwEnASMdPDK2l1ETAE2BcYCdwD/Jj10v0X6XTkN2MjBK7PaIuI90iydR5F6\n9s4h/X/zEPAtYNuBGrwC98AyMzMzMzMzM7N+zj2wzMzMzMzMzMysX3MAy8zMzMzMzMzM+jUHsMzM\nzMzMzMzMrF9zAMvMzMzMzMzMzPo1B7DMzMzMzMzMzKxfcwDLzMzMzMzMzMz6NQewzMzMzMzMzMys\nX3MAy8zMzMzMzMzM+jUHsMzMzKxhkqZJCknnN/EcY/M5QtJyzTqP9T1JYwo/24+3uj3dIenI3N5H\nJKnV7Wk2SXubbIAiAAAQhklEQVTm631J0hKtbo+ZmbUvB7DMzMxaSNLwwgN8T5bhrb4Ws8EuB1DP\nyLvfj4ioUmcRSZtJ+g9Jl0l6XNK8VgRhJW0v6SRJ4yU9Juk1Se9Keiu361eStio7RkRcB0wCVge+\n1ScNNzMzq2JoqxtgZmZmZjZAnACsCDwGXFWjzg+BsX3WonJXAB+s8vrCwAZ5OVzSpcCREfFejeOc\nBlwNHCfpwoj4V1Naa2ZmVsIBLDMzs9Z6BdiopHwCsBrwKvDpOsfpcxHR9N4kEXE+0LQhimbdIWlZ\n4Ji8e2613leVqoXtd0i9l9YC1mhi82qZCdwE3AM8DUwBpgMrA5sCR5ICXIcC7wJH1TjOn4BngQ8D\nxwEnN7XVZmZmVaj2/71mZmbWapImkx5+X4iI4a1tjVnvkjSG1LMH4BMR8VAr21NG0nHAOcAsYJWI\neKtGvb2ADwD3AY9HxDxJ1wB75SrLR8S0Pmrz0IiYW1K+NHAnKYgewNoR8UKNuqcCpwBvAGtExKwm\nNNnMzKwm58AyMzMzM6vv8Ly+rlbwCiAi/hQRF0fEIxExr4/aVqstNYNXuXwG8NO8K2DbkupX5PWK\nwJiet87MzGzBOIBlZmY2SEg6PyeJnpb3V5R0qqSHCrMFHlKov4ykg3Ki6UdzYuf3JL0u6RZJX5e0\neJ1z1pyFsNoMc5IOlHSbpH9Jmi3paUlnS1qx5BylsxDm64vcy6WSGP8CSc9KmiXpDUkTJO3Rzc9x\nv3z9b0iaKelJSWdU2tibMy9K2lnS5ZKek/SOpBn5Z3GupJpDzjp/JpKWkPSfkh7MP8e3JE2UdLSk\nId1oxzKSTpZ0T77uOZJekXS1pM8uwPXsIemKfD0z83FeyMc5PPf4qXeMBb5H8vvWlHRW/gym53v5\ntZy8/A+SDpO0fHevpdOxPwasn3dr5b5qCknr5t/tR/N1zZb0fL5vtuiFU8wobC9Wq1JEPA08mncP\n6IXzmpmZLRDnwDIzMxuEJG0E3ED1BM4VdwAbV3l9GLBjXr4qabeImNzDJi0i6Wq69twYARwP7CNp\nu4h4uScnkTQK+COwbOHlxYDRwGhJ342I02u8dyHg18BBnYo+Qsr5c5CksjxkC9LOJYFxQLXg0IZ5\nOUrSIRHxP3UOtwrpmjfs9PrIvOwvadeSIW9bANcCK3UqWo308xoj6UZgv4h4u8YxhuU2bF+leM28\njAGWonY+s4bvkfxzuQpYslPRSnn5KPA5Uk6o39c4f5kdCtv3NPD+hkj6DvA9uv7NPjwvB0k6Gzix\nJCdXPZ8vbP+jTt27ScMNd5S0aETMafCcZmZmC8w9sMzMzAafhYHxpDw85wI7A5sB+wNPFeoNISWY\nPpWUn2cksHWudw0pJ876wFXd6cVTx3mkwMSVwN6kBNK75vNAehj/WQ/P8aF8/HeAbwLbAJsDxwKV\nWdNOlbR5jfefTkfw6nlSguuRwKeAs+kIFPXoC8D8WV5HR/DqWuCLpOFbWwJfB54DFgd+K+mTdQ45\njhS8+g3pM92M1ENmYi7fGvhDjbZ8CLiZFOQJ4BLSZAGb5TY9kKvuAlwpSVWOsTTwNzqCV/eRPrtt\n8nHGAD8CXqxzHQ3dI5KWAX5LCl5NA87M7d0U2Ip0P18AvFTn/GW2y+upEVHvOnqFpO+TZv8bCjwM\nfJX5f5fvyFVPYAGSqksaImk1SbvkwOSeueg+4K46b6/cU0uQfrfMzMz6TkR48eLFixcvXvrpAkwm\nBRYmd6Pu+blukGYU27ZO/RF1yj9bON4+NepMy+XnVykbU3h/AGNrHOOqXP4+MLxK+djCMZarUv5Q\nofwpYOUqdTYB5uY6l1Yp/zDwXi5/gpRou3Od0cC8wrm6XHM3f6bH5/fPBkbVqLM0KaAQwCPkiXdq\nfCYBHFPlGENIwbFKnf2q1JlQKP9SlfKhwJ8LdQ6uUueXhfJzOre107FW7e17pNN9ul3J5z4EWLbB\nn9mr+fjXN/j+ayi5h6vUH1m4175T7TMl5az6Wa4zB1itzjHnMv9nXVweIyVwr9eujQvvObaRz8KL\nFy9evHhpdHEPLDMzs8Hpooj4e1mFiHimTvl44Pa829OkzbdHRK2hYz/KawH1ehvVc0REvNb5xYiY\nBPwl71Yb5nYYHT2rvh4Rb1Y5xk2kIYYNk7QwKYAFcE5E3FytXqTk2sfm3Y1IvW5quTMiLqhyjHnA\nV0iz5gEc3aktI0hBOYBrI6LLtUVKAn4YqVcbpN5hxWOsChySd+8GToiIqkPZImJuREwpuY5G75FV\n8nouaUa9qiJiXkRMLzl/VZKGFs7R5d5qkhNJIyXuiojTq32m+bXjgenAIsCBDZxnNnAMsHlEPN+N\n+q8Xttdp4HxmZmYNcwDLzMxscPrNgr5B0ko5YfSGlQWYmour5crqrfZMIvXogJ49FL8cEX8tKb8/\nr4dXGQq3c16/CtxacoxxjTYuGwmsmrf/WKfu3aReYZCGwtVyaa2CHMy7Pu9uLWmJQvGowvZ/lxxj\nKqknF8CmnRKhf5o0ZBVSj7RG8zBB4/fIq3k9lDTssbetSAqcAfy7CcefTw5y7pp3S++RiHiHjvu6\n7B6B9Du8EfBx0s/+NOBt4IfAOZJqJnAvKF7/KjVrmZmZNYEDWGZmZoNPZdhZXZJ2yjPETSP1LnmK\nNNNYZflCrvqBHrbpyVoFEfEu6UEa0tC5Xj9HVnn4FimZeFElAfqkOkGY+0vKuqPYk+rBwkyCXRbS\ncLlKcKgsWHBvnXNWyocCGxReLyZ9r5eYvFKuTu/bpLD9tzrHqKfRe+QmoNKz6zJJd0o6UdJWkhbp\nYZsgBbAqpvXC8erZgJRjCuC8snsk3yc75bqlAaWIeDwiHouIhyPiLxFxCim5/fOk/Fq3Slq0zjHm\n0NEbr3PCfDMzs6ZyAMvMzGzwmRkR79WrJOk80rC6Mcw/a181i/ewTe/UKX8/r3uSLL6755jvPLnn\nSaX3yT/LDhBpFr7ZDbUu6TzTX3ctUVL2ekkZzD/srRiMWSGv36cjyX0tUwvbKxS2h+V10PPhdQ3d\nI7kX0u5AZUjs1qReRXcB0yRNkPTFPBSwEcWAZnd6KfVUM+6RqiLideDwvLsVHcNWq8o9FytBrrr/\nxpiZmfWmHs2iY2ZmZv3SvHoVJO1DSgQOqefLuaQH/pdJAbB5ud4FpLxHXWafs4YUgy+j6eg5VE9Z\ngKknw/bSAXo29K/lIuJBSRuQZtTbk5Qn60OkwOvovJwgafeIeHkBD1/sdbVCzVq9p3iPfJuO4Zv1\nzKpfpauIuFvSP0nByH2Bs0qqL1toX1/0RjMzM/t/DmCZmZm1p6/k9avAlhHxVo16ffHA3lIRMVvS\nbFLvmmFldSUtRc964RQDUW9GxGM9OFbFypT3wlq5sP1GYbsypHIhScMioqz3WXF4WjEPUuV6lM/T\n3YBcr8sJ56/OSyXB/C7AUaTcYx8j5TDbcQEPPZU0y9+iwPJ16vaG4j3ybi/dI/VUAlhr1alXvP4X\nm9ccMzOzrjyE0MzMrD19NK9vLAleQfnsd4PJ43m9SZUE70U9/TweLGxv28NjVYysU755Xs9j/jxT\nxcDIFnWOUSmPTu97oLDd0xkke1VETImIS0lDCu/OL+8gaYHyueXeaZXPbd1ebGItTwDv5u3eukdq\nyvd7ZWKBt8vqAusVth+vWcvMzKwJHMAyMzNrT5Ve2DUTMUv6FPM/sA5mt+T1apT30Dm4h+e5k45e\nUF+tlzS7mw6pVSBpJVJ+KIC7ImJmofgvhe0v1znGZ/LuAxHxZqF4Ah25kI6tE/xriTwc9vbCS41M\nSDAxrzfs5mx9Dcs5vW7Ou7tLGtHM85Fm4Kz0rHq0Tt1ioHNizVpmZmZN4ACWmZlZe6okvB4labXO\nhZJWBy7u2ya11MV05A67UFKXoWKSRgFf6slJImI28KO8uy5p1ryaM+VJWlLS2Frl2baSvlblvUOA\nX9CRgP+iTm15mjSDH8Dekr5AJznx+cV0BDov7HSMKcCv8+5WwNm1gliShkgqnSmvEZJ2lFRz6Fu+\nhh3y7nukPG8LqhJQWhj4RAPvX1BnkHq7DQXGS/pgrYqSFpK0b+dAl6Sd6wW/JK0D/LLw0rg67ar0\n9ns6Il6oU9fMzKxXOQeWmZlZexoHbEfKcXWXpB8AD5ESNG8PfANYDriX+kPUBryIeFrSOcCJwPrA\nA/kzeZAUANqDNEPbk8BwUkCn0cTnZ5N6eY0GvgBsLukXpM96OrA0qefb9sBepNnlzi853n2koNtI\n4ApSDqURpCT9W+Y6N0fEH6q892vApHzOKyTtAPwReDO34Rt0DJu8Cbi8yjGOz20dARwHbCfpV6Te\nPO+SerVtA+yfr6PsWhrxGeBoSbcDNwKPkD6DxXObjqDjc7g8zyS5oCaQZklcAtiJjiGJXUhagY4e\naxVrFrYPkFSccfGJiLi3WDknVj8FOA3YEHhc0i+BW0mzPS6Wj7kVsA+wOun3+ZnCYbYFJki6FbiB\n9PN4g/QF9pqknlcH0xGcHB8R40uua1E6hjReXauemZlZsziAZWZm1p4uISW43oeUuPlnncrfBY4k\nJb4e9AGs7GRgDVKgZW1S76WiV4HPAffk/dmNnCQi5knai9Qj6suk2fLKZn57s6QMUq+wK0nBiGpD\nHCcC+9Voy7OSRgN/AlYiBXuOqFJ1ArBvtdkKI+ItSZ8ExpMCKpvTkXerrwwhBZZ2KqlzAykIucAi\n4m1JVwMH5uX0kuprApeWlF/Uaf/HpOBl53OeLmkaqcfessA381LNXFKArbOFSIGqnUva8z7p9/+4\nkjqQgrjL5O1qgUwzM7Om8hBCMzOzNhQR75OCGkeQAjJvkwIyz5OGjI2MiEta18K+FxHzIuIAUq+o\n24BpwCzgaVKvqU2Ap+josTK9B+eaHRGHkYajXURKjD6dNIxxOqkX0WWkYNrqdQ43hRRk/G5+3wzS\nz/N+UsBm24io2daIuIfUU+k7pEDKm6ShdlNIga19ImKXsp5LETGV1MtqX+Aq0jC9OXmZTApuHcr8\nw9V6yynA50kBx3tJs+PNzstkUo+yvSJitwZ7X1VUAk8fkbRJD47TbRHxE1Iw9VTgLlLPsrnATFJv\nq6uBo4E1ImJSp7efR7qXf07qMTaZFOSaQ+rFdQfwX8AGEXF0RLxLuQPz+vaIcAJ3MzPrc6ryRZqZ\nmZmZVSFpQzoSXe8XEVe2qB1jSQEKgOUjYlor2tFuJP2NNIzupxHRJe/YYJVzl00GFgV2j4jrW9si\nMzNrR+6BZWZmZtZ9++d14FnY2tEpeX1YWWL1QehEUvDqHgevzMysVRzAMjMzMwMkLSNpWEn5NqSE\n5ZCGUb3UNy2z/iIibgOuIwVzTmpxc/qEpJWBo0hB2+PrVDczM2saJ3E3MzMzS9YEJkoaT0pa/gwp\nF9TqwO7AIcAipBxEtZJp2+B3DGnmxtmSVC2x/SCzFmmSgZcj4q5WN8bMzNqXc2CZmZmZ0SW/VS2z\ngINblfuqwjmwzMzMrN24B5aZmZlZ8gxwALArsCkwDFieNKPfc8DNwIUR8UrLWmhmZmbWptwDy8zM\nzMzMzMzM+jUncTczMzMzMzMzs37NASwzMzMzMzMzM+vXHMAyMzMzMzMzM7N+zQEsMzMzMzMzMzPr\n1xzAMjMzMzMzMzOzfs0BLDMzMzMzMzMz69f+D2Yt23B2BHAjAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1440x1152 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xIJwGIIr0SH7",
        "colab_type": "text"
      },
      "source": [
        "Potential Challenges:\n",
        "\n",
        "1) Can you install a policy of exploration from http://incompleteideas.net/book/bookdraft2017nov5.pdf (Sutton and Barto, 2017), Chapter 1. The policy of exploration we have implemented is an e-greedy policy but can we do better?\n",
        "Exploration/Exploitation is a key problem within RL. Consider: how this implementation will affect the workflow of code; Is the policy defined within the agent's class methods or do we keep it within the environment? Alter the code to implement appropriately.\n",
        "\n",
        "2) Detail the parameters, which govern the efficacy of learning. How do they influence the learning update? \n",
        "\n",
        "3) Code and implement a Q learning and SARSA agent instead of a Monte Carlo agent. Again, the algorithm/learning update is detailed in Sutton and Barto, above. \n",
        "\n",
        "4) The reward alllocation scheme we have implemented is far from optimal. Can you identify the reasons why? How may we improve the reward allocation scheme? Implement your improvements and quantify the improvements observed in learning. Note some reward allocations allow for online learning, think about how the code structure may change.\n",
        "\n",
        "NB: Sutton and Barto are renowned for their contribution to RL. They are a good port of call for any learning or questions you may have. I found when I got stuck on these problems it was usually that I hadn't quite got the entire concept. Their writing is really clear and logical, try to spend some time understanding each chapter.\n",
        "\n",
        "Another great source is David Silver's lecture series on RL at UCL. He is one of the people who developed the AlphaGo and AlphaZero algorithms at DeepMind. In parallel to this work, try to read a few articles on state of the art RL - they will expand your knowledge and perception of this work. I have recommended a few in the literature list.\n",
        "\n",
        "\n"
      ]
    }
  ]
}