Analyze_ab_test_results_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze A/B Test Results\n",
    "\n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "For this project, I will be working to understand the results of an A/B test run by an e-commerce website.  My goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.\n",
    "\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### Part I - Probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. Read in the dataset and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in dataset & view top rows\n",
    "df_ab = pd.read_csv('ab_data.csv')\n",
    "df_ab.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Use the cell below to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "294478"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ab.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df_ab.user_id.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "d. The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Proportion of converted: 0.11965919355605512\n"
     ]
    }
   ],
   "source": [
    "converted = df_ab.converted.value_counts()\n",
    "print(f\"Proportion of converted: {converted[1] / (converted[0] + converted[1])}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "e. The number of times the `new_page` and `treatment` don't match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first = df_ab.query('group==\"treatment\" & landing_page!=\"new_page\"')\n",
    "second = df_ab.query('landing_page==\"new_page\" & group!=\"treatment\"')\n",
    "len(first) + len(second)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Do any of the rows have missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         1965\n",
       "timestamp       1965\n",
       "group           1965\n",
       "landing_page    1965\n",
       "converted       1965\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.isna(first).count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id         1928\n",
       "timestamp       1928\n",
       "group           1928\n",
       "landing_page    1928\n",
       "converted       1928\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.isna(second).count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.`\n",
    "a. Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>857184</td>\n",
       "      <td>2017-01-20 07:34:59.832626</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>327</th>\n",
       "      <td>686623</td>\n",
       "      <td>2017-01-09 14:26:40.734775</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>856078</td>\n",
       "      <td>2017-01-12 12:29:30.354835</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>666385</td>\n",
       "      <td>2017-01-23 08:11:54.823806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>748761</td>\n",
       "      <td>2017-01-10 15:47:44.445196</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     user_id                   timestamp      group landing_page  converted\n",
       "308   857184  2017-01-20 07:34:59.832626  treatment     old_page          0\n",
       "327   686623  2017-01-09 14:26:40.734775  treatment     old_page          0\n",
       "357   856078  2017-01-12 12:29:30.354835  treatment     old_page          0\n",
       "685   666385  2017-01-23 08:11:54.823806  treatment     old_page          0\n",
       "713   748761  2017-01-10 15:47:44.445196  treatment     old_page          0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = first.copy()\n",
    "df2.append(second)\n",
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df2.append(second)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Misunderstood and put all of the bad rows into DF2. I will see about removing all of the DF2 rows from df_ab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make copy so as to keep df2 intact\n",
    "df_mess = df2.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>308</th>\n",
       "      <td>857184</td>\n",
       "      <td>2017-01-20 07:34:59.832626</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>327</th>\n",
       "      <td>686623</td>\n",
       "      <td>2017-01-09 14:26:40.734775</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>357</th>\n",
       "      <td>856078</td>\n",
       "      <td>2017-01-12 12:29:30.354835</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>685</th>\n",
       "      <td>666385</td>\n",
       "      <td>2017-01-23 08:11:54.823806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>713</th>\n",
       "      <td>748761</td>\n",
       "      <td>2017-01-10 15:47:44.445196</td>\n",
       "      <td>treatment</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     user_id                   timestamp      group landing_page  converted\n",
       "308   857184  2017-01-20 07:34:59.832626  treatment     old_page          0\n",
       "327   686623  2017-01-09 14:26:40.734775  treatment     old_page          0\n",
       "357   856078  2017-01-12 12:29:30.354835  treatment     old_page          0\n",
       "685   666385  2017-01-23 08:11:54.823806  treatment     old_page          0\n",
       "713   748761  2017-01-10 15:47:44.445196  treatment     old_page          0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_mess.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove the rows that match df_mess"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = pd.concat([df_ab, df_mess, df_mess]).drop_duplicates(keep=False)\n",
    "df2.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290585"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the correct rows were removed - this should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` a. How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2.user_id.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "b. There is one **user_id** repeated in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2_dup = df2[df2.user_id.duplicated()]\n",
    "df2_dup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is the row information for the repeat **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#see above"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Remove **one** of the rows with a duplicate **user_id**, but keep your dataframe as **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.user_id == 773192]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.drop(1899, inplace=True)\n",
    "df2[df2.user_id == 773192]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` a. What is the probability of an individual converting regardless of the page they receive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all rows that have a successful conversion\n",
    "df2_conv = df2[df2.converted == 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34753"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many records\n",
    "all_convert = len(df2_conv)\n",
    "all_convert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# how many total transactions\n",
    "all_peeps = len(df2)\n",
    "all_peeps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Probability of conversion regardless of page they receive\n",
    "all_convert / all_peeps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all control rows\n",
    "df2_all_control = df2[df2.group == 'control']\n",
    "all_control = len(df2_all_control)\n",
    "all_control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17489"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all control rows that converted\n",
    "df2_control_conv = df2_all_control[df2_all_control.converted == 1]\n",
    "len(df2_control_conv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get percentage of control transactions that converted\n",
    "df2.query('group == \"control\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get percentage of treatment transactions that converted\n",
    "df2.query('group == \"treatment\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2[df2.landing_page == 'new_page']) / all_peeps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Consider your results from parts (a) through (d) above, and explain below whether you think there is sufficient evidence to conclude that the new treatment page leads to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Considering the results from parts (a) through (d) above, there is NOT sufficient evidents to conclude that the new treatment page leads to more conversions.__The control converted percentage is 0.12 and the treatment converted percentage is slightly LESS at 0.118. They essentially have the same conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### Part II - A/B Test\n",
    "\n",
    "`1.` For now, consider you need to make the decision just based on all the data provided.  If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should your null and alternative hypotheses be?  You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the converted rates for the old and new pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Null:* uP_new - uP_old <= 0\n",
    "\n",
    "*ALT:* uP_new - uP_old > 0\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have \"true\" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. <br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. What is the **conversion rate** for $p_{new}$ under the null? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We are looking at a null where there is no difference in conversion based\n",
    "# on page, which means the conversions for each page are the same.\n",
    "\n",
    "p_new = df2.converted.mean()\n",
    "p_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. What is the **conversion rate** for $p_{old}$ under the null? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We are looking at a null where there is no difference in conversion based\n",
    "# on page, which means the conversions for each page are the same.\n",
    "\n",
    "p_old = df2.converted.mean()\n",
    "p_old"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show that there is no difference in conversion based on page\n",
    "p_new - p_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. What is $n_{new}$, the number of individuals in the treatment group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new = len(df2[df2.group == 'treatment'])\n",
    "n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. What is $n_{old}$, the number of individuals in the control group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old = len(df2[df2.group == 'control'])\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null.  Store these $n_{new}$ 1's and 0's in **new_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_page_converted = []\n",
    "new_page_converted = np.random.choice([0,1],n_new,[1-p_new,p_new])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null.  Store these $n_{old}$ 1's and 0's in **old_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "old_page_converted = []\n",
    "old_page_converted = np.random.choice([0,1],n_old,[1-p_old,p_old])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Find $p_{new}$ - $p_{old}$ for your simulated values from part (e) and (f)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0013557785262576783"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#new_page_converted.mean() - old_page_converted.mean()\n",
    "obs_diff = np.mean(new_page_converted) - np.mean(old_page_converted)\n",
    "obs_diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_style('darkgrid')\n",
    "\n",
    "#mpl.style.use('ggplot')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Create 10,000 $p_{new}$ - $p_{old}$ values using the same simulation process you used in parts (a) through (g) above. Store all 10,000 values in a NumPy array called **p_diffs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = np.array(0)\n",
    "new_page_converted = []\n",
    "old_page_converted = []\n",
    "\n",
    "for _ in range(10000):\n",
    "    bootsamp2 = df2.sample(n_new, replace=True)\n",
    "    new_page_converted = np.random.choice([0,1],n_new,[1-p_new,p_new])\n",
    "    old_page_converted = np.random.choice([0,1],n_old,[1-p_old,p_old])\n",
    "    diff = np.mean(new_page_converted) - np.mean(old_page_converted)\n",
    "    p_diffs = np.append(p_diffs, diff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "i. Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAo0AAAJuCAYAAADVfThhAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3XeYZGWZsPF7AgMzDMMQRoISVPRBUUFcw+e6goJZBAEjLgyurmFRdxUVFQXFCEYUcc2AOYAKKrsGwmLAhLAiPCoSFEFxYYBhyPT3x3uKrqmp6lOnp7qruuf+XVdf3XVSPed01VtPvenMGRsbQ5IkSZrI3GEHIEmSpNFn0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFrzhx3AdIuI3YEzgMszc/vhRrO6iDgT2A04ODM/27Z8d0Y0Zhj9+EZBRGwJvB14ErAl5b13VmbuPsy4pF4iojUf270z87JhxjJbRMT2wKUAmTlnuNFMHcu72WvGJI0R8VngoI7FdwA3ANcBvwF+BnwxMy+dppiWAv8OkJlHTsdzTreIWA5sD3wjM3893GhmpoiYD/wQeEC16DrgNuDaPvffHHgc8A/Aw4GHAUuq1Qsz85Y+jrGA8lp9PrAD5b1zEfAZ4BOZOekJW9f22BGxJ/Aq4JGU87oSOA14Z2b+tWbfLYE3AE8H7glcTykHPpiZP5jsOWn2s2ybGqNQ3nU55jeAvauHJ2Tm8h7bXQZsV3O412bmexs+/wLghcAjgF0oifTmwC3AH4D/Ao7NzKtqjvMPwKHAY4FNgb8B/w28OzP/ULPvEuB1wH6Uc7wZ+DVwfGZ+rd9zmTFJY5vbGX/xzaG8mDYF7kt5Ubw9Ir4OvDwzr+my/yogKR9Ma2spcET195EDON4VlNiuH8CxBmU5pfbzMsoLrJtBXtPZ6EmUAvRa4FGZ+fuG+78A+MBkn7wqLH5IKXyh/L8WAo+qfvaKiGdm5h3TfeyIeBOlRgLgLmAlcB/glcDzIuLxmfmbHvs+pHruzapFN1AK4qcDT4uIN2bmu5uek9YZy6kv2wbtdkpZOZsNtbzrFBF7M54w9quV6HZz0yTC2BQ4vu3xnZTyainw0OrnZVVZeUa3A0TEQcAnKXnbWLX/NsC/AM+NiGdk5g977Hsv4Gzg3tWilZTc6fHA4yPiY5n5sn5OZCb2afxxZm5Z/WyRmQuBTYCnAF+mXMz9gfMi4p6dO2fmzzJzx8zcY3rDrpeZB1axnTLsWJoY5Ws6Inaqfp8xiQIUymv6z8ApwJuANzbc/xOUpO5aYC9gMbCI8qF5CyXJeusk4lqrY0fEUxlPGN8HLM3MjYEHUT7ElwHfjIj1u+y7EPgWJWE8D3hQte8m1bHmAO+KiCdO8rykgcvMK6uycsdhxzKFhl3e3S0iFgMfpiRYFzfYdd+2PKPz5/j63ddwC/BBYF9Ki8iCzNwU2AB4ahXbxsBXqxbMzvN4CKWsnQ98HtgiM5dSasq/B2wIfD0ilnXZdw7wNUrCeBnwj5m5EbARpebxLuClEfHifk5kJtY0riEzVwCnA6dXzdinUP4xX6fUdkjDtLD6vXKS+38kMz/UelD1Ie1LRDwUeHb18ODMPK36+07ghKqA+iDwHxHxocz82zQe+53V729k5qGthZl5YUTsRWnivg/wr5SCv91LKE0sK4G9MvPKat8bgEMj4r7APsC7KM03kqbH0Mq7Lo6i1Ma9ipKwDSVZr3KU/+iy/DbguxHxO0oz9WaUL98ndWz6NmA94BfAQZl5Z7X/5RGxL/BbynkeBrymY9+9KV1/7gKe2eqKUTXzHxMRW1O6F70tIk6oYuppViSN7TLz9Ig4FPgI8MiI2CszT22tn2jQRkTMBQ6k9J18CKX6dgWl38C5wFcy8/Rq2zMpTRutfTv7bb211c+xfYAL8A1KH6y9gW2B26pvDD0HwnSqPlBfTekbsR7wv8CHM/MLXbbdnpqO192uSdXf5zNtm30mItoft2+7xv5dnuNxwCuAR1Oq6q8FflLF3atK/e6O+JQX/JspNcrLgL9QvhS8rUoUOvddALwMeA7wQMo3sWuBq4FzgM9l5k+6Pe9EmpxHl364B1VNDC19DTBoFRCT9Pzxw+S3uqz/OKUmcGNKofqx6Th2ROwE7Fw9PLpzx8z8c0R8EXgxcABrJo0HVL+/0EoYOxxDSRp3jYgdM7PvWobJvHYi4pHV8z2OUngvo5QdvwI+2avPUNtr5K2UJPp11bltB/ydUl4ckZnXVds/jFL78mjKdf0N8I7M/EaXYy+nvIfPyszdq9fey6pzGqN8AB3TKtOaqmpxXgk8E7g/sD7wJ+C71XH/1GWfvsvYPmNYADyBcu0fQaksWAz8lfK/en9m/rJjn+X0Wbb18fyNzqdXeRwRRzLe1WkiXQeTVJ8JL6IkB5tWMZxLScD+q59z6WYGlnft8exKif3XwHGUMmgkZeYlEXEdpaVk6/Z11Zfvp1YP3995fTJzZUR8DHgHpUvPoR39yFtl5fd79N19LyWp3pLSXD3h+28mNk/34xOUQgPGP9j6cRKlMNmd8ga5iVIIPJCS8B3Ztu21lEK95a8dP92+ZS0Dfkn5YNieMmCgkYh4FaVZrpWwtvqPfT4iOj9Y18bNlPO4vXp8A6ufX7f+ol1FxNspfc+eCdyDcl3vQSnofxAR76o5xM6UJsgXUf4fcynX7zXV/ut1PN98Su3SB4H/V+2zkvIt7iHAyylvkkYmcR7XU65Vqw/MLax+DQdSONZ4XPW7a21bZt4M/E/18PHTeOzWvtdTPty6aX3YPaJKUACIiI0Y70PZ6wPxp4z3De77vCbz2qli+ynlW/4jKe/zW6rfT6I0Of1nzVMvAL5PqRnZntK8vg3lQ++/I2KDqm/Wjyivtw2qn38ATo6IZ3c7aFuMHwA+SxlYcCelaerxlFqOQyfYtdfxHkCVsFYxLKKUZztUMZ8fEf/YZdcmZWw/nkgZNPUiyv9nISUh3pZS9v80Iv65Y5+BlW0DPJ+VrPkZ0v7TtfYnItaLiM9RPhOeAWxBOb9llK4hp0fEGl/K+jFDyzvg7mT+PymfFS8fVCI6VSIiKAkjVF8q2jyGUjkEvVtNWuXgVowPQGrZvWOb1VRfui+sHtaWlbMyaayqV1vfgv6pn30i4rGUQuYuSjXykqoGcANK5r+c8s219Rz7Ugrg1uPOfg/dRle9hfLPfwqwKDOXUArcfi2j1KCcCGyVmZtQOv6/r1p/SEQ0SZJ7yswvZ+aWwI+rRa/qOL+HT7R/S0Q8l1IzAqX29x5V3MsYrz06LCJeMMFhPkv5tvjg6potpnT+vZVy/Tr7YjyfklSvAv6Zcq03odSEbAccApzfT/xrcx6Z+arqGrZeC1/uuIZr1MQMUtWXpdUcc+EEm/62+v3AaTx26/FFmXlXzb7tzwWlUGzV0nR97uqYrQEHfZ8Xk3vt3AV8B3gepaZrg+p1ugklgVoJ/GtEPGuC5305cD/KB/2GlNf4PsCNlNf4EcAJlP5MW1dl0z2Ab1KuxQerhLebXSjNT+8BNq3O557VsQCOjojHTHxZxkXExtX5bkepCd2VMqp1MaVV4KTq3L/e3j+raRnbp5WUpG0PYPPM3DBLP/ftKIn/fODjEbFta4cBlm0DO5/MfG+vPnSUz4tW0vPdjl2PptQkXVbFslGWvr0bUbpw3AC8NiKe108cbec248q7DodQ3jefnkyLEvCBiLgmIm6LiKsj4jsR8fyImDeoACNibkRsFRHPAb5dLb4COLVj01b5dXVm/l+Pw/227e+7y7uIuAclR4ABfQbMuubpNv9LVYhHxHqZeXvN9q2+j/+dmR9sLayqea+iFNhra33gqdk2GjRrhsl3WETp9Lq8Vf1cNVsdGmWagoOAt0bEF3MtplAZlCqxOKp6+KXMfEVrXfXif2UV9/Moo96/0COBuJJy3W6t9r0V+HTVp+4QysCnj7Zt3/pfnpiZn2t7zjspb8rjhnQe020JJQGB0pzfS2vdVtN47K061k+0b+f+W/XYpt/nnkjj105mrgKe1mX5CuAjEXE95Yvey4Gv9njejYG9M/OstmXfjIhjKP2ZDqMMLPiXtuNfExEHUMqnrShNiGf3OPYnM/Owtn2vqmrgtqbU+h4J7Nkjtk6vpdSGfpMyYODusqZqfjwwIjajNKm9iPEkYuBlbGaeCZzZZfkVlL60SyhTnRzM5Ad79TLlnxnVh/43KDWoX8nM97Stux+le8AKYI/M/GNbDCspyfIKygDRNwFf7PM5Z2p5B0CUAbBvB/4PeP0kD7ML5YvjLZTa26dUP/8aEftU7+3JxvdJSqVHp18Dz65aaNrVlpWZeXP1v17KFJeVs7KmsXJd29+b9rF9q1/cPaqq7anw3ewxfUgD7+qREL6j+r0D433Fhm0XSjwwPkq2U6sg347SJ6mb97cSxg6tflwP6lje+l82SRYmMqjzmG4btv3dWRC1W1X9XjzBNoM+dmv/fvbt3H8qz2vQrx0Yrzl41AQ1FT/pSBhbvt/29xrdODLzJkrTOKz5Pmj3zs4FVTnSOubjI6KfchLG+619YIIvp60E5Qlty6ajjO3UuvbdmsrX1pSeT9Xt5muUpvbzKIlvuwMpn+HfaE8YO5xMaZHZKSL6fU3P1PKu5VhKTethE9TM9fINyjyGrVrrJZRzfC+lRnk34CtrGV+rGb89R/k18IrsPtq8n7ISupd3Ay8rZ3NNY7t+at2+T+k3sitwZkR8HPhhZk6UnTc1mWrydrdT+jStITN/HxGtGoddmb55xyaya/X7mszs1YyYEXElpblsV8Y/ANv9vMfxWwMgNulY/l3KN8y9I+JblObtsyZRgLQM6jymW/vAp0HXPK/tsVv7T7Rvr3VTeSeNSb12qqbhg4BnUb60bUrpp9huA8pr9e+s6X97HLp9xHmvL5yt/tud74OWK7L3DQ/OoTR9zqMkC10HpbVExDbAvaqHX42IXjVMrXPfpm3ZlJSxVbL7b5SaoKDUrHYm51t37jcAU/2Z8WFK96q/AftUNdrtHl393j8injLBcVr94bah1IDWmanlHRHxdMqAl58Cn2q6f2b+e5dlV1Ca+C+ltDQ8ISKemJmTmpUhM19DNcK5qgl/KvBu4H8i4n3ZNpNEpZ+yspeBfwbM5prG9gL0up5bVapm4pdRsvF/ovTLuTIiLo2I46um0LXVpIN1N3/PiYfDt5KoNeZqGpJWHHWTfv+5Y/tON/ZY3rozwGpffqoam7dQOubvRRll/feIuCgi3ls16zQxqPOYbu2DsRZNsF1rXZMpMtb22Cs71nfT/i15ZY+/F9Jb4/OazGunGghzFmXi3dZt0+6kvN9bgwBaNuzcv9Lrw/zuDvzZ+24RrW3W67G+5+u2agprlY/9vG7ba6uWUZruuv20yt+7/79TUcZGxAMp/bHeRhm4tCml1uRvrF6b0+u6T9pUfmZExMspfRJvA/arEpdOrf/FYnr/H7Zg/HN+ovdauxlZ3kXEhpSk7k7K4JdBf1E+ntJ3FErZsNYy84bM/BKlJvwG4DUR8cyOzfopK9vX9yorB/IZMJuTxgdXv//cR39GADLz05SO3P9O6a/zf5S+Oy8FfhkRk55ktDLVI7hG9V6ma0zOPNUy8yjKNCBvoIwau4EymOI1wG8j4sBJHHbaz2Mt3cD4SMaJalpa6/qphRjUsf/SsX6ifTv3/0uPbfp97glN4rXzZkqtz98ptY1bZOaizLxHNTCg/SYDo/gebRJT+2fGxpk5p+Zn+/adp6CM/QwlMfoV8GTKQJAlWW78sCWl5rfpOfZtKj4zImI3yiAegH/LzF6DaVr/i1f18X+YU/X/bGKmlXevozTlfxr4fUQsbv9hvPZ5ftvyvl8XVRLaavW6zyADzzKCuXVTjxd2rK4tK6Pc7KA16GxKy8pZmTRGmburdXeS/5lo206Z+dfM/FBm7kP5BvUIyj9zDnBUlJnZh2Xz6tx6aX3zbK/RvHtan4jYoMd+G69tYD204th2wq3Gm7vWtiZ2NZl5aWa+OzOfTKmBeBxloMB84KNVJ/N+DPU8Jqsq5C6qHu40waatEXO/nWCbQR+79fgBE/QHa+3b/lxQ7p7QqkXo+tzVMaPHc9dq+NppJSavyMwTc81JzLdo+vwDNtGHzQaMf9j087ptrzVtMir9boMqY6sR0Y+gfBl/Rmb+VzUApN2UX/tBfmZExHaUwVLrUeZY/OQEm7f+F5P6P0xgRpZ3jN8z+sWU1qnOn9YMAQe0Lau7z3SntWkqrtOq2b1vx/JW+bVlNcCsm/bXwN3lXZZbKbe6wwzkM2BWJo2UF02rUP/8RBtOJDPHMvPnlA+FP1OuV/vUFHf352nyjWUtrEdpgllDROzA+IfDr9pWtY/yuhfdTTTFROscJ3N+rTg2jIiunaUj4v6M18T8qts2g5CZd1bftJ9O6Ru6If1PdzQy5zEJrfuYPqHbyippaE1L9YNpPHZr343p/fpr3QLw3GrABwCZeSNlYuqez02ZL7H1Zajpea2mj9dO6311Xo9D9DsqeapsF2VS6W4eQ6mBGaOPftBV38hWsrLWkyX3UcZO5O6kJbtP8A4TX/u1Kdu6WpvziYhFlNrKZZT3xxp3EOnQ6iO/V3TMVbuWZnJ5N2Wqz/jW+/6yKXiK9vtCtzuH8TlFe72eW2XlVaz+BRvqy+l7Mp5Q1paVsy5pjIgnUeYyhDIi8dsTbd+2X88avCzTbbT+ae1V9u13IlnjfpFT5A09EtQ3VL9/T9s8ctU378uqh2vctL365vKiCZ6vdY6TOb9fU26NBL3vH3pk9fsy4GeTeI411NTG3sZ4N4F+m1+Gch4D0hrFumPVSbzTiynJ1c2MN49M+bEz87eMv05f27ljlFtbteaW6/bFr3X3owN6jAptdSb/ZWZml/VdTfK105pE/MF0qJrF3tS5fAje0LmgKkda0/D8IDOv7fNYn61+vzzKJN9dRcScKHM6th5PpoydSOu6b9Gt1SAiHszEN3dYm7JtKs7nM5RBVJcCz8rMups/nEBJfLemy/+3I9Zeg6S6mZHlXWYun6h5ntLvGOCEtuWXtfbvo+LnJZSuBzA+r2Jfovccqq3196PMywodraOZeT1lXlSAV3e2zFR9OV9aPfxCl76crbLyiRHRbWaVV1O+OF3FeILZ06xIGiNi44h4UpTbjn2H0jn+T5T5+/r1zoj4WkTs0z71RERsERHHUr4FjFHmSQTunoet1WegczqEqbCKMmP7p1qFZEQsjYj3MN4P4sguL5rWFAGHR8QzWi/giHgUZQTgRB+UrdFz+7Z/APSjiuPw6uHeEfHhVvV6RGxWXddWYnB4Dm6urxMj4jPVa2Kj1sKqtuUEyijW9ruVjOp5tOKeGxGbt35YvTvBZh3rOmM/j/H//2cj4qnVMedVffNa8759oEuzKhHx2YgYi4jLBn1sxj+Q9ouIo1v/q2pww6mUaTP+SLnDU6f/BC6vtjmt2oeI2CjKHTBatWBN+5RN5rXTKhPeHxG7tT58IuLhlG/ua/xfptkNlPnl3tl6D0fElpTz2YNSrjWZw/DdlP/LhsBZEXFQrH7Hnm0i4sWUu1+1d+pvXMbWuIhSmzcH+HLV2tK6S8q+1XEm6tg/6bKtMrDziYjDKPdxX0lpaq+d6SEzL2K87+NbI+K4iLi7r12UPntPiIiT6D0/aLfjztjybi0dGxEfiojHROkj2Ipnm4h4N2WScyjzpXZOsk5ELK/KyrFYs2b/2Ig4NiIeHW3dxKrP7+WUhHYhpcn8A11iO4LyJeQRlLJ282r/bSnTKm1LaVV8T5d9v0m569Zc4JTqc5+IWD8iXkPpjwvldqUT3ncaZuaUO4+OiKvbHm/E6qOCxigfZP+Wmd2mtuhlPmV+pv0AIuIGSmG0Uds2h+ea8yx+kjLa8n0R8TbG+w98MNsmfB2QayiFxAeA5VEm89yY8eT/uOxy/2lKIb8/pfPuN4FbI+IOSqF/BWWC2M4bpLecRKm1eQxlFOnfKC/eP2dmbbNLZn45yjf+N1Em4n55lMmO2+N+d2ZOuhtBFxtQ7hu8HBirnm8B46+TO4GXNHl9DOk8WrZlzVtLtfy543G3b8svpvSTeRjw7YhYRWmSbNWAnEZ/973tZtLHzszvRMSbKRMJv5byLbp1GzYo76W9s8scnVkms92bkpTtClxYvWcXU/4fY8Abs/m0GJN57RxOafrZhjLR9C0RcSfl/XUzpQZh0vf/HYDzqp83AK+rrtNSxl8rr5tgsMUaMnNFlBadb1HuzvNZymT7KygffO0j2tu/wE62jO0Vx10R8UrKXIa7UwY/3Eh57S2glG2HMkVl24DP58nV7/WA70dEr+1+nOVuZC2vo1zvl1Emj395dQ3upJRNrf/xmX3EcLcZXt5N1kaUgWyvBO6qznce4+URlOSuSWVUy6Lq2K9oO/YcVq/lvppSw9x5jmTm+dUXsU9S7lT1gur11kqob6KMsl+jf2lmjkXE/pQ+2fcGfhIRKyllXSsH/FhmdvtyvoaZWNO4HuNTCWxOmbj0j5QC7E3AfTPzOQ0TRiiJ2CspSdXvKP/Q9Sk1ll8GHpuZa0yQS5nq4fXABdU+21U/U9JcXSWiz6C8eOdSpp35KfCCzDykxz7XUUZ3fpxSMzqXMsrvw5QP3DVepG37Xkz5QDyd0hy0JeX8evWP7HaMwyk1Gt+kJAKLq+f/FrBnZk7YtDIJh1EK09Mpr40FlDf/JZQmoF0zs9cHSU9DOI+ByMwbKP//wyhNwmOU981PKU0uz5igKazV9Nt1rsy1PDaZ+XbK6+vblOlR1qf8z44FHjTRB25mnk+Z0PrYap/1Kf+PbwNPyMx399p3Ao1fO1kmVn4E8DnKVC/zKN/6Pw88fBKJ68Bl5n9QEuFfUj4oVlKaop6S3W95Wne8PwAPpSQqZwDXUj5c76CUhR+mTITcfq0mW8ZOFMcplNaX71Fqadaj1EC/t4pvKsu2gZ9Ptf9E0+esNgF71d/25ZTE93OUc19ASSSvoHQLOYjxps++zdTybi18jPK6+THlc3IDxv+fp1Bqgh8/QTeOVll5JWuOQn43pWz5b0qTfuuL6F8pX3xfDew40Ze3zDyBMqbhK9V+rRbVTwO7ZGbPOVarRHQXyiT/F1PKgBsp791nZ+bLeu3bac7Y2NDvNidpBEXpxnAdpcZsl8y8YMghqYGq2eszlMnJdx9uNNLsFhGnU+ZpfUVmfqRu+5lqJtY0SpoeD6PULpxiwihJ3UW5PeijKTWUE02TNOOZNErq5bHV76OGGoUkjbZdKX0ij87MW+o2nslm4kAYSdMgM49hfPoqSVIX1dyco3i3p4GzplGSJEm1HAgjSZKkWut88/Q119y4TmTNixevz8qVa0x3py68Vv3zWvXPa9U/r1V/vE79m03XatmyjYbWFG7z9Dpi/vx5ww5hxvBa9c9r1T+vVf+8Vv3xOvXPazUYJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRa84cdgCTNRidfcNVqjxctXMCqm28bUjRrZ9+HbDXsECSNAGsaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbXmDzsASWo5+YKrhh2CJKkHaxolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJteYP88kjYn3go8CewKbAH4A3ZuZ3I2J74FLgprZd3pOZR7XtezywP7AKODoz39927D2A44BtgXOB5Zl5+ZSflCRJ0iw01KSxev4/AbsBVwBPBb4SEQ9u22ZpZt7RZd8jgfsB2wFbAmdExG8z8/SI2Bw4GXgRcCpwFPBl4FFTdSKSJEmz2VCTxsy8iZL8tZwWEZcCDwN+WbP7gcDBmXkdcF1EfAJYDpwO7AtcmJlfBYiII4G/R8SOmXnxQE9CkiRpHTDsmsbVRMQWwP2BC9sWXx4RY8D3gNdm5t8jYhNga+D8tu3OB/ap/t6pfV1m3hQRl1TLV0saFy9en/nz5w38XEbNvHlzWbp00bDDmBG8Vv0b9LVatHDBwI41aubOnTNjz2+63w++B/vjdeqf12owRiZpjIj1gM8DJ2TmxRGxGHg48GtgM0r/xM8DTwIWV7td33aI64GNqr8XA9d0PEX7+rutXHnroE5hpC1duogVK1YNO4wZwWvVv0Ffq1U33zawY42aRQsXzNjzm+73g+/B/nid+jebrtWyZWukMtNmJJLGiJgLnATcBhwCkJkrgV9Um/w1Ig4BroqIJcDKavkS4Ja2v2+s/l5ZPW7Xvl6SJEkNDH3KnYiYA3wK2ALYLzNv77HpWPV7TtWP8Spg57b1OzPerH1h+7qI2BC4L6s3e0uSJKlPo1DTeDzwAGDPzLy5tTAiHgmsAH4PbAIcC5yZma0m6ROBwyPiF5SE88XAwdW6U4BjImI/4NvAW4ALHAQjSZI0OcOep3E74CXArcDVEdFa9RLgLuCdwD2AGygDYZ7XtvsRlITzcuBmyhyOpwNk5jVVwvgR4HOUeRqfO9XnI0mSNFsNe8qdy4E5E2zyxQn2vRV4YfXTbf33gR3XKkBJkiQBI9CnUZIkSaPPpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklQCzVf2AAAgAElEQVTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUy6RRkiRJtUwaJUmSVMukUZIkSbVMGiVJklRr/jCfPCLWBz4K7AlsCvwBeGNmfrdavwdwHLAtcC6wPDMvb9v3eGB/YBVwdGa+v+3YPfeVJElSM8OuaZwP/AnYDdgYeDPwlYjYPiI2B06ulm0K/AL4ctu+RwL3A7YDHge8LiKeDNDHvpIkSWpgqDWNmXkTJflrOS0iLgUeBmwGXJiZXwWIiCOBv0fEjpl5MXAgcHBmXgdcFxGfAJYDpwP71uwrSZKkBoZd07iaiNgCuD9wIbATcH5rXZVgXgLsFBGbAFu3r6/+3qn6u+e+Uxm/JEnSbDXUmsZ2EbEe8HnghMy8OCIWA9d0bHY9sBGwuO1x5zqq9b32Xc3ixeszf/68tYx+9M2bN5elSxcNO4wZwWvVv0Ffq0ULFwzsWKNm7tw5M/b8pvv94HuwP16n/nmtBmMkksaImAucBNwGHFItXgks6dh0CXBjta71+JaOdXX7rmblylvXJvQZY+nSRaxYsWrYYcwIXqv+Dfparbr5toEda9QsWrhgxp7fdL8ffA/2x+vUv9l0rZYtW6P+a9oMvXk6IuYAnwK2APbLzNurVRcCO7dttyFwX0pfxeuAq9rXV39fWLfvFJ2GJEnSrDYKNY3HAw8A9szMm9uWnwIcExH7Ad8G3gJc0DaQ5UTg8Ij4BSXhfDFwcJ/7SpIkqYGh1jRGxHbAS4BdgKsjYmX1c0BmXgPsB7wDuA54JPDctt2PoAxuuRw4CzgmM08H6GNfSZIkNTDsKXcuB+ZMsP77wI491t0KvLD6abSvJEmSmhl6n0ZJkiSNPpNGSZIk1TJplCRJUi2TRkmSJNUyaZQkSVItk0ZJkiTVMmmUJElSrb7naYyITYCtgEuqORJbyw8G9gFuAj6YmT8beJSSJEkaqiY1je8Ezm3fJyJeAXwS2Ityx5UzI+KBA41QkiRJQ9ckafxH4Acd94c+FLgSeCzw7GrZqwcUmyRJkkZEk9sI3hP4QetBVaO4DfD6zDynWvYsSgIpSZKkWaRJTeNC4Ja2x/8IjAHfb1t2CSW5lCRJ0izSJGm8Etix7fGTgBuA89uWbQK0N19LkiRpFmjSPH0GcFBEHEKpcXwG8PXMvKttmx2APw0wPkmSJI2AJjWN7wJWAh8CPk5JHI9srYyIewC7AT8eYHySJEkaAX3XNGbmpRGxE7B/tehbmXlF2ybbAccBXxhgfJIkSRoBTZqnycyrgY/0WPdz4OeDCEqSJEmjpVHS2BIRGwL3BxZn5v8MNiRJkiSNmkZJY0Tci9KncS9gHmXKnfnVusdQ+jq+PDPPHGyYkiRJGqa+B8JExFaU2wjuDZwG/ASY07bJucA9gOcMMkBJkiQNX5PR00dQksI9M3Nf4HvtKzPzduB/KJN+S5IkaRZpkjQ+lTJi+swJtrkC2HqtIpIkSdLIaZI0bgH8vmab24ENJx+OJEmSRlGTpPFaYJuabe4PXD35cCRJkjSKmiSNPwKeERFbdlsZEfcDnky53aAkSZJmkSZJ4zHABsBZEfEUYBGUORurx6cCdwHvG3iUkiRJGqomtxE8NyL+FfgYZcqdlhuq33cAL8zMCwcYnyRJkkZAk5pGMvMzwIOAY4GfAZcAvwI+CjwkMz8/8AglSZI0dI1vI5iZvwf+YwpikSRJ0ohqVNMoSZKkdVPPmsaI2HayB83MKya7ryRJkkbPRM3TlwFjkzjmWM1xJUmSNMNMlNydyOSSRkmSJM0yPZPGzFw+jXFIkiRphDkQRpIkSbUm1fcwIrYBHgpsDFwPnJeZfxpkYJIkSRodjZLG6v7SHwUe32XdD4F/y8zfDSg2SZIkjYi+k8aI2AH4MbAZ5U4w5wBXA1sCjwH2AM6JiEdn5h+mIFZJkiQNSZOaxndREsZXAcdl5l2tFRExF3gF8AHgncCzBxmkJEmShqtJ0rgH8J3M/HDniiqB/FBEPBHYc1DBSZIkaTQ0GT29APh1zTa/BtabfDiSJEkaRU2SxvOBHWq22QG4YPLhSJIkaRQ1SRrfCewbEU/ptjIingY8E3jHIAKTJEnS6GjSp3Ez4LvAaRHxA+Bs4K/AFsBulGl4TgU2j4gD23fMzBMHE64kSZKGoUnS+FnKvajnUAa7dBvw8gxgr7bHc6p9TBolSZJmsCZJ48FTFoUkSZJGWt9JY2aeMJWBSJIkaXQ1GQgjSZKkdVSje09LktY9J19w1bQ+36KFC1h1820DP+6+D9lq4MeU1iWNksaIeATwemAX4F499h/LTJNRSZKkWaTv5C4i9ge+RGnSvgz4GXDH1IQlSZKkUdKkRvBI4CbgaZl5ztSEI0mSpFHUZCDMDsAXTRglSZLWPU2SxquB26cqEEmSJI2uJknjV4EnRMSCqQpGkiRJo6lJ0ngEsAL4SkRsN0XxSJIkaQQ1uSPMqoj4V+AM4I8RsQK4vsumY5l530EFKEmSpOHru6YxIh4D/BjYBLgTWAXM6fLjXWYkSZJmmSZT7rwHWA84EPhCZt41NSFJkiRp1DRJGnemTLnzuakKRpIkSaOpSVPySuDaqQpEkiRJo6tJ0vgdYLepCkSSJEmjq0nSeBiwJCKOi4gNpyogSZIkjZ4mfRq/BNwIvBQ4MCJ+R+8pd/YYRHCSJEkaDU2Sxt3b/t4QeGiP7cYmHY0kSZJGUpPJvZ1/UZIkaR1lIihJkqRaJo2SJEmq1aRP490i4l7APYH1u63PzLPXJihJkiSNlkZJY0Q8EfgAsGPNpvMmHZEkSZJGTt/N0xHxSOA0YCnwEWAOcDbwCeDi6vGpwNsGH6YkSZKGqUmfxjcCtwAPz8xXVcvOyMyXAg8CjgL2BL422BAlSZI0bE2Sxv8HfCsz/9K5f2aOZeYRwEXAWwcYnyRJkkZAk6RxY+CKtse3USb5bvcj4LFrG5QkSZJGS5Ok8W/AJh2P79uxzXrAwrUNSpIkSaOlSdL4O1ZPEn8KPCEi7g8QEVsC+wG/H1x4kiRJGgVNksbTgd0iYtPq8YcotYrnRcTPKSOolwEfHGyIkiRJGrYmSeN/Uvor3g6QmT8CngVcShk9fRXwssw8cdBBSpIkabj6ntw7M28Azu1YdgpwyqCDkiRJ0mjx3tOSJEmqNal7T7dExDOAx1PdHSYzvz6QqCRJkjRSJqxpjIi9IuLsiNity7rPUJqmXwm8AvhKRJg0SpIkzUJ1zdPPAHaloy9jRDwdOAhYBbwdeD3wR2CfiHjeFMQpSZKkIaprnn4E8JPMvKVj+QuBMeDgzPwaQEScBFwCHAB8cdCBSpIkaXjqahq3pCSCnR4LrADubo7OzKuBbwMPHVh0kiRJGgl1SeMmwLXtCyJiW2BT4JzMHOvY/lJgs8GFJ0mSpFFQ1zx9I3CvjmUPq36f12OfzqbsCUXEIcBy4MHAFzNzebV8e0oSelPb5u/JzKOq9esDxwP7U/pWHp2Z72877h7AccC2lD6ZyzPz8iaxSZIkqahLGv8XeFpELM7MldWyZ1L6M57TZft7U+4M08RfKINpnkS5LWGnpZl5R5flRwL3A7ajNKOfERG/zczTI2Jz4GTgRcCpwFHAl4FHNYxNGnknX9D0LTc4ixYuYNXNtw3t+SVJ06euefrzlCbqsyLilRHxEcpAl6uBM9o3jIg5wGOA3zYJIDNPzsxvAP/XZD/gQOCozLwuMy8CPkGpsQTYF7gwM79aDeI5Etg5InZs+BySJEmivqbxU5QE7EnALpRJvG8HXpWZd3Zsuwelxu/7A47x8ogYA74HvDYz/x4RmwBbA+e3bXc+sE/1907t6zLzpoi4pFp+cfvBFy9en/nz5w045NEzb95cli5dNOwwZoSZdq0WLVwwtOeeO3fOUJ9/JvFa9W+qrtVMel/3Y6aVVcPktRqMCZPGzLwrIp4GPA94NKU28OTM/HWXzTcHPgR8a0Cx/R14OPBryuCa4yg1n08CFlfbXN+2/fXARtXfi4FrOo7Xvv5uK1feOqBwR9vSpYtYsWLVsMOYEWbatRpm87DN0/3zWvVvqq7VTHpf92OmlVXDNJuu1bJla6Qy06b2NoKZeRclWft8zXZfAr40oLio+lD+onr412rAzFURsQRo9a9cwvjAmyWUgTtU65d0HLJ9vSRJkhqo69M4SlrT+8zJzOsoA252blu/M3Bh9feF7esiYkPgvm3rJUmS1EBtTeNUi4j5VRzzgHkRsQFwB2VqnxXA7ymDcY4FzszMVpP0icDhEfELYAvgxcDB1bpTgGMiYj/KhONvAS7IzNX6M0qSJKk/o1DTeDhwM3AY8ILq78OB+wCnU5qUfwPcSulb2XIE5W41lwNnAcdk5ukAmXkNsB/wDuA64JHAc6fhXCRJkmalOWNjnTd1Wbdcc82N68QFmE2dgKfaTLtWztM4M3it+jdV12rfh2w18GMO00wrq4ZpNl2rZcs2mjOs5x6FmkZJkiSNOJNGSZIk1eqZNEbEtRHxurbHb4mIx05PWJIkSRolE9U0LgU2aHt8JLD7VAYjSZKk0TRR0vhX4F7TFYgkSZJG10TzNP4U+OeIuJMykTbA7hFRd8yxzDxqEMFJkiRpNEyUNL4WuD/wkrZlu1PfRD0GmDRKkiTNIj2Txsz8Q0Q8GLg3cE/gTOCzwAnTEpkkSZJGxoS3EczMuyh3Xbmkapa+LDPPmo7AJEmSNDr6vvd0ZjqnoyRJ0jqq76SxXUTcC3goZVqe64FfZeafBxmYJEmSRkejpDEitgU+Djyhy7rvAS/NzMsGE5okSZJGRd9JY0RsCfyIMijmMuBsylQ8WwGPAZ4InBMR/5CZVw8+VEmSJA1Lk5rGN1MSxtcD78/MO1srImIe8B/A0cDhwCGDDFKSJEnD1SRpfBrw35l5TOeKKoF8b0TsCTwdk0ZJkqRZpcmI6C2BX9Zs88tqO0mSJM0iTZLG64HtarbZttpOkiRJs0iTpPEcYP+IeHS3lRHxSOBZ1XaSJEmaRZr0aXwHpV/jWRHxJeAMyujpLSn3o34ecBfwzgHHKEmSpCFrckeYX0XE/pT7Tx8APL9t9RzgWuCFmVnX71GSJEkzTKPJvTPztIjYDtgb2BXYmNKH8TzgG5l50+BDlCRJ0rA1vo1glRh+ofqRJEnSOqDJQBhJkiSto0waJUmSVMukUZIkSbVMGiVJklTLpFGSJEm1TBolSZJUq++kMSJ+GBFHTWUwkiRJGk1NahofBcybqkAkSZI0upokjb8HtpmqQCRJkjS6miSNnwSeFhHbTlUwkiRJGk1NbiN4KvAE4EcR8R7g58DVwFjnhpl5xWDCkyRJ0ihokjT+kZIgzgE+NMF2Yw2PK0mSpBHXJLk7kS61ipIkSZr9+k4aM3P5FMYhSZKkEebk3pIkSao1qb6HEbEj8ABgcWaeNNiQJEmSNGoaJY0RsQtl6p2Hti0+qVq3G/Bd4DmZeerAIpQkSdLQNbmN4P2BM4GgjJ7+bscmZwPXAvsPKjhJkiSNhiZ9Go8AFgCPyMxXU+ZpvFtmjgE/AR4+uPAkSZI0CpokjXsAJ2fmRRNscwWw9dqFJEmSpFHTJGlcCvy5j+MtmHw4kiRJGkVNksa/ATvUbLMT8KfJhyNJkqRR1CRp/CGwV0REt5UR8XBKE/Z/DSIwSZIkjY4mU+68C3gWcHZEHEnVdzEidgIeSxkocyPw3gHHKEmSpCHru6YxMxPYj9Jn8SPAi4A5wAXAcdXyfTPziimIU5IkSUPUaHLvzDw9Iu4NHAQ8CtgMuB74KfCZzLx28CFKkiRp2BrfRjAzV1Am9/7Q4MORJEnSKGoyEEaSJEnrqMY1jRFxAPBCyv2nlwA3AOcBn87Mzw82PEmSJI2CvpPGiFgP+BrwdMoAmDuAayj9Gh8H7B4Rzwb2z8zbpyBWSZIkDUmT5uk3AHsB51KSxA0ycytgA+DxwM8oCeXrBx2kJEmShqtJ8/SBwB+A3TPzttbCzLwLODMidgd+AywH3j7AGCVJkjRkTWoa7wV8sz1hbJeZtwLfBO45iMAkSZI0OpokjX8B1qvZZr1qO0mSJM0iTZLGLwD7R8SSbisjYimwP+AIakmSpFmmSZ/GtwEPAn4WEW8Dzgb+CmwB7Aa8mTIY5qhBBylJkqTh6pk0RsRdwFiXVXOAk3osvx9w80THlSRJ0swzUXJ3Nt2TRkmSJK1jeiaNmbn7NMYhSZKkEea9pyVJklTLpFGSJEm1Gg9YiYi9gF0ok313m7dxLDP/ZW0DkyRJ0ujoO2mMiO2A04AHUkZK9zIGmDRKkiTNIk1qGo8FdgI+DZwIXAncMRVBSZIkabQ0SRofD/xXZr5oqoKRJEnSaGoyEOZ24H+nKhBJkiSNriZJ448otxGUJEnSOqZJ0vgW4LER8dypCkaSJEmjqe8+jZl5XkTsAXw7Il4C/Aq4vsumY5l51KAClCRJ0vA1mXJnY+BdwKbAbtVPN2OASaMkSdIs0mT09AeA3YHvAycBf8EpdyRJktYJTZLGpwM/zswnTlUwkiRJGk1NBsIsBH48VYFIkiRpdDVJGs8D7jNVgUiSJGl0NUkajwL2iojHTFUwkiRJGk1N+jRuBZwG/DAivgD8ku5T7pCZJw4gNkmSJI2IJknjZynT6cwBDqx+xjq2mVMtM2mUJEmaRZokjQdPWRSSJEkaaU3uCHPCVAYiSZKk0dVkIIwkSZLWUSaNkiRJqtXk3tN/7HPTscy87yTjkSRJ0ghqMhBmLmuOlgbYGFha/f0X4Pa1DUqSJEmjpclAmO17rYuIHYBjgQ2BJ619WJIkSRolTWoae8rMP0TEvsBvgCOAN/S7b0QcAiwHHgx8MTOXt63bAzgO2BY4F1iemZdX69YHjgf2B1YBR2fm+/vZV5IkSc0MbCBMZt4CfA94XsNd/wK8Hfh0+8KI2Bw4GXgzsCnwC+DLbZscCdwP2A54HPC6iHhyn/tKkiSpgUGPnr4D2LLJDpl5cmZ+A/i/jlX7Ahdm5lerhPRIYOeI2LFafyBwVGZel5kXAZ+g1Fj2s68kSZIaGEjzNNxdu/dM4E8DOuROwPmtB5l5U0RcAuwUEX8Ftm5fX/29T92+wMXtT7J48frMnz9vQCGPrnnz5rJ06aJhhzEjzLRrtWjhgqE999y5c4b6/DOJ16p/U3WtZtL7uh8zrawaJq/VYDSZcuctExxjG2Bvykjqvvsz1lgMXNOx7Hpgo2pd63Hnurp9V7Ny5a1rHehMsHTpIlasWDXsMGaEmXatVt1829Cee9HCBUN9/pnEa9W/qbpWM+l93Y+ZVlYN02y6VsuWrZHKTJsmNY1H1qy/AXh7Zh49+XBWsxJY0rFsCXBjta71+JaOdXX7SpIkqaEmSePjeiy/C7gOuDgz71j7kO52IXBQ60FEbAjcl9JX8bqIuArYmTL4hurvC+v2HWB8kiRJ64wm8zSeNRUBRMT8Ko55wLyI2IAyoOYU4JiI2A/4NvAW4ILMbPVJPBE4PCJ+AWwBvBg4uFpXt68kSZIaGIV7Tx8O3AwcBryg+vvwzLwG2A94B6Um85HAc9v2OwK4BLgcOAs4JjNPB+hjX0mSJDUwZ2ys250Bi4iYVFKZmXdNOqJpds01N/a+ALPIbOoEPNVm2rU6+YKrhvbcDu7on9eqf1N1rfZ9yFYDP+YwzbSyaphm07VatmyjOcN67rrm6cncR3qsj+NKkiRpBqlL7v5ESQL7sRjYbO3CkSRJ0iiaMGnMzO3rDhAR6wGvAN5ULbpsraOSJEnSSFmrgTAR8SzgIuAYYA7wOuABA4hLkiRJI2RSfQ8j4tHA+4BHUKbHORZ4W2ZeN8DYJEmSNCIaJY0RsQPwbso9pucAXwMOy8w/TkFskiRJGhF9JY0RsSllXsSXAAuAnwCvycyfTmFskiRJGhETJo0RsQD4d+ANwMaUybQPy8yvT0NskiRJGhF1NY0JbAtcS0kej8vMO6c8KkmSJI2UuqRxO8o8jXOAQ4FDI6LumGOZud0AYpMkSdKI6KdP4xxg0+pHkiRJ66C6yb3Xah5HSZIkzQ4mhZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRa84cdgCRJ0+HkC64adggDs+9Dthp2CFoHWdMoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSapl0ihJkqRaJo2SJEmqZdIoSZKkWiaNkiRJqmXSKEmSpFomjZIkSao1f9gB1ImIM4FHAXdUi67MzKjWPR94F7A58D3ghZl5bbVuU+BTwBOBvwNvyMwvTG/0kiRJs8NMqWk8JDMXVz+thHEn4D+Bfwa2AFYBH23b5zjgtmrdAcDx1T6SJElqaORrGidwAHBqZp4NEBFvBi6KiI2Au4D9gAdl5krgnIj4FiXBPGxYAUuSJM1UMyVpfFdEvBtI4E2ZeSawE/Dj1gaZeUlE3Abcn5I03pmZv2s7xvnAbtMXskbdyRdc1XX5ooULWHXzbdMcjSRJo20mJI2vB35LaWp+LnBqROwCLAau79j2emAj4M4J1q1m8eL1mT9/3qBjHjnz5s1l6dJFww5jpCxauKDr8rlz5/Rcp9V5rfrnteqf16re0qWLLNcb8FoNxsgnjZl5btvDEyLiecBTgZXAko7NlwA3Umoae61bzcqVtw4u2BG2dOkiVqxYNewwRkqv2kRrGvvnteqf16p/Xqt6K1asslxvYDZdq2XL1qj/mjYzZSBMuzFgDnAhsHNrYUTcB1gf+F31Mz8i7te2387VPpIkSWpopGsaI2Ip8EjgLMqUO88BHgv8OyX2n0TEPwG/At4GnJyZN1b7ngy8LSJeBOwC7A08etpPQpIkaRYY6aQRWA94O7AjpZ/ixcA+mZkAEfFS4P+3d+9RcpRlHse/YyAaQEDAC8hNBR9ZXAQFXcFIVFhdj654OWddUQkosnAUWRQQFYVFYUVZAyLK4Qjoys3lIirexSxo1AVlBREeLxhFF4QEEiWJJIbZP97qUGl6pnou3TM9fD/nzOmZqnqr3nqnpus3b1W9fSGwJfBt4OBa2SOA84C7gaXA4ZlpT6MkSdI4TOvQmJn3AHuNMv8ioOOA3dUg3wf0qGqSJEmPKIN4T6MkSZL6zNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSZKkRoZGSZIkNTI0SpIkqZGhUZIkSY0MjZIkSWq0wVRX4JHgny64AYBL5+/Z9fKXzt9zvXLt62gt06lc07pvX7qS69/1wq7L7L3gOhYdNXddHVruWLaK7Tafs678Xqdfu269ey+4ju02n7PecncsW8WatcM8dcuNuHT+nux1+rVsOGuI7Tafw+1LV/LULTfijmWr1tvGoqPmstfp1wJw/bteuG69ty9dyYazhgBYs3Z4ve8BNpw1tO77bswaGmLt8DCzhoZgCNY+2H3ZsW5DkkbS7fvEqd/65YTWu9XGs1myYnV5z4P15s0aGuJxG20IwJIVq9lq49nct3INa4eH2Wrj2QDct3LNunKzhoY4dr+dOHfRb9crs2TF6vqEoCsAAA9oSURBVHXbum/lGo7dbycATvv2rwDW/fya3bYG1j/X1M978NB5cKRzYOvn1jnijmWrWHTU3PXaoL7e9vV1WudYjPU8P6gMjX1w+9KV41q+Xq59HZ3W2c12ullPu1b4Gss216wdHrH+9e/ry3Vbl9Zy9VDYHhDHEhjhoTfMtcPD0KNcZ2CU1KRX7xPt620Fuk7bWzs8vG5+fdn279vX3W2Zkfax/Vwz2vlqpHnt555Oy3Sz/rGaSNlB4uVpSZIkNTI0SpIkqZGhUZIkSY28p1ETdsVNd3b8vtsyk71uSZI0+explCRJUiNDoyRJkhoZGiVJktTI0ChJkqRGhkZJkiQ1MjRKkiSpkaFRkiRJjQyNkiRJamRolCRJUiNDoyRJkhoZGiVJktTI0ChJkqRGhkZJkiQ1MjRKkiSpkaFRkiRJjQyNkiRJamRolCRJUqMNproCkiTpkeuKm+582Pf1aSNNH2mZTj9vNGd2x/U0rUPrm9GhMSK2AD4D/D2wBDg+My+aqvqM5WAc6Q9jtD+Y0bax0ZzZrFy1uuv1dLteSZL0yDDTL09/ElgNPBE4EPhUROw6tVWSJEkaPDM2NEbExsBrgRMy8/7M/B7wJeBNU1szSZKkwTM0PDw81XXoiYjYA1iUmXNq094N7JuZr5y6mkmSJA2eGdvTCGwCLG+bthx47BTURZIkaaDN5NB4P7Bp27RNgT9PQV0kSZIG2kwOjb8ANoiInWvTngXcMkX1kSRJGlgz9p5GgIi4BBgG3grsDnwV2DszDY6SJEljMKPHaQSOAM4D7gaWAofPpMA4lnEoI2II+HdKgKYqd1xmDlfzd6+m7QLcCrwlM/+3Vv7ZwALg2cAK4JTMPKMX+9UL/WyrapnZwE3AJpm57eTvUW/0q50i4hjgIGCHajtnZ+ZHe7Vfk6Xb9plg24xadhD0qZ0G8hhq14+2qq1jIN+XWvrVVoN+vuulmXx5msy8NzMPyMyNM3P7qRzYu0fGMg7l24ADKJfodwNeARwG695IrgI+DzwO+CxwVTWdiNgK+DpwDrAlsBPwzd7sUs/0pa1qjqH8szJo+tVOQ8Cbq3kvA94eEa/vxQ5Nsm7bZyJtM2LZAdKPdhrUY6hdP9qqZVDfl1p63lYz5HzXMzM6NM5k4xiH8iDg9Mz8fWb+ATgdmF/Nm0fpdV6QmQ9k5pmUN+QXV/OPBr6RmRdW8/+cmbf2ZMd6oM9tRUQ8BXgjcGoPdqdn+tlOmXlaZv4kM/+amUl5E9+nR7s2KcbYPhM5hkYrO+31q50G8Rhq18djamDfl1r62FYDfb7rNUPj4Ho6sDYzf1Gb9lNgpF6hXav5nZbdFbip7fLXTbX5fwfcGxGLIuLuiPhyRGw/4T3on362FcAngPcCqyZS6SnQ73YC1l1Kmsv0f0htLO0zkbYZrewg6Fc7rTNAx1C7frbVoL4vtfSrrQb9fNdThsbBNdZxKNuXXw5sUr3ZNq1rW8p/bu8Etgd+A1w87pr3X9/aKiJeDWyQmVdOtNJToJ/HVN2JlPei88dY334byz5NpG1GKzsI+tVOdScyGMdQu7601YC/L7X067ga9PNdT830B2EGVkQsBPYdYfb3gXcwtnEo28et3BS4PzOHI6JpTMtVwJWZeX1Vt5OAJRGxWWa2//H13XRpq+ryyWnAy7uvff9Ml3Zqq9PbKfelzc3MB5r2YYqNZezXibTNiGXHW/E+61c7AQN3DLXreVtN9/elMejXcTWtz3dTzZ7GaSoz52Xm0AhfL2Ds41DeUs3vtOwtwG5tPRm71ebfRBm6qKX1/bTo+ZhGbbUzsCNwXUTcBVwBbB0Rd0XEjhPczQmbRu0EQEQcArwHeElm/n5ie9cXY2mfibTNaGUHQb/aaRCPoXb9aKtp/b40Bv06rqb1+W6q2dM4oDJzRURcAfxbRLTGoXwVsPcIRT4HHB0RX6X8EbyLco8LwEJgLXBkRHwaOLSafk31ej5weUScSfnDOgH4XmYum9y96o0+ttWDwHa19ewNnEUZtuGeSduhHunnMRURBwKnAC/KzNt7sDuTboztM5G/t9HKTnv9aqdBPIba9amtBvp9qaWPf38Dfb7rNXsaB9sRwBzKEAoXUxuHMiLmVt3wLecAXwZuBn4GXF1NIzNXU4YneDOwDDgEOKCaTmZeQ7mB+upqWzsBb+j1zk2ynrdV9RTnXa0v4F7gwerntX3Zy4nryzEFfIgynMX1EXF/9fXpXu/cJOjYPpPcNiOWHSD9aKdBPYba9bStZsj7UkvPj6sZcr7rmRn9iTCSJEmaHPY0SpIkqZGhUZIkSY0MjZIkSWpkaJQkSVIjQ6MkSZIaGRolSZLUyNAoSV2KiBMjYjgi5k3R9udX25/fNn1xRCyeijrV6jClbSOp9/xEGOkRLCJaA7UOAztn5q9HWO67wLzqx4Mz84Le1643OnwG91rK587+kfIRYl8D/isz73946Qlv+wLgIOApmbl4stffS1VQPZ8B//1LGj97GiX9lfK5qm/pNLP6rNd9q+Vmks8CJwEfBi4AbgP2A84DfhkRL+9Q5ixgF+B/+lTHdldW279yirY/mqluG0k9Zk+jpD8CdwIHR8QHMrM9HL6VEiq/Qvn4rZnigsxcWJ8QEY+hfE7tvwFXRsT+mXlta35mLgGW9LWWNZm5HFg+VdsfzVS3jaTeMzRKAjiX8tmsrwC+2JoYERtSLqcuAm5hhNAYEVsAx1TzdwRWAzcAH8nMb7YtuxnwNuAfgKcDT6AEoR8Ap2TmDzusfxj4b+B1wCnAK4EtgF8BH8vM88e32+vLzL8AH46I2cAHgDOAPWr1OBH4IPCieuCMiLnAsdWyjwfuAxYDX8vMk2r70PKbiGh9/9vM3LFaZiGlV/fRwHuAAynteXFmzm+6RFy17YeBV1M+l/l24NPAJzJzuLbcPOC7wEmZeWKH9Syu2qO9XgDnR0S9vZ+SmYtHapuq/Esox8dzgY2A3wFXAKdWQbi+bGtbG1La9GBge8rnAF8EnFD7/GlJfeTlaUkAFwMrKL2Kdf8IPJESKjuKiB2AH1NCzj2UkHIp5VLl1yPi0LYiu1CCzYPA1cB/AN8CXgxcFxEvG2FTmwPfB54PXAZ8DtgGOC8iDupqL7v3MWAVsHtE7DraglV9FwIvAL4DnE4J3g8AR9QWPQn4afX9GdXPJwELOqz28qrsomr+zV3UeTbwbeClwCWU39nm1bbO6qL8aC4Arqq+v4qH6n4SsGy0ghFxGOX3uw+lXRYA9wLHAYsiYvMRil4EvAO4DvgU5fdxLOWfG0lTwJ5GSWTmnyPiEmB+RGybmb+vZh0K/An4AvDeEYp/FtgB+OfMvKQ1sQoDC4EzI+JLmfnHatatwDbV5Uxqy29LuR/u48DXO2znWcBngMMyc21V5uOUh1eOq+oxKar2+DElCD6X0ss6kkMp/4DPy8yf1mdExFa1dZ4YETtW+7Gg4UGYHYBntrdRg60pPYvPzMwHqu1/ELgeOCIiLq1fah+LzLyg6hl9FfDFbh+Eqf6hOBO4H3huZt5Wm3c2cDhwGqXnud3TgF0z895q+fdRQvebI+L4zLxrPPsiafzsaZTUci4wCzgE1p3w9wcuzMyVnQpExLMolxIvrwdGgMxcRrlc+RjgtbXpyzuFoSqoXgY8IyK277C5lcDRrcBYlfk5pfdxl4h47Bj2tRt/qF4f3+Xyq9onjDH01Z0wzrLHtwJjtf17gZOrHw8eZ10m4o2UHtCz6oGx8j7KU+tviohHdyh7XCswAmTmCuBCynlrzx7VV9Io7GmUBEBm/igibgYOiYgPUS5VP4pRLk1TLhUDbFbd09auFbh2qU+MiH2Ad1bln0AJFnVPptz3VvfLzPxTh23cUb1uTgkhk2Woeh0edakSZF4D/CgiLqXcK/j9Wm/teIznCeS/Ui5nt1tYve7RYV6vPbt6vaZ9RmbeFxE3Ai8EnsFDl+5bbuiwvtbv+nGTVkNJXTM0Sqo7l3I58WWUnqkfZ+aNoyy/ZfW6f/U1kk1a30TEqyk9in+h3Ov2a8r9lA9SxoJsPQjSbqR751pPe88aZfvjsU31es9oC2XmFRHxCspT14cAhwFUl7ePz8xvjWPb47n0uqTeC9thXZuNY50T1drmnSPMb01/2H2NVU91u179riV1wdAoqe4/gY9QHjZ4MmXomdG0nnx9Z2ae2eU2TqY8Xb1nZt5anxER57D+wNtTorrU/Zzqxx81LZ+ZVwNXR8TGwPMoT6EfDnwlIvaoLqN3rf6k8xhsFRGzOgTHJ1Wv9aeUH6xeRzoHbMbkDO3TWseT6Hxf6NZty0maxrynUdI6Ve/OZcC2lN6/ixuKtIbHmTuGzewE/LxDYHwU5cGT6eAYYA5wY3s9R5OZKzLzmsw8mjI00GzK0EItrUDXi56yDYC9O0yfV73We4zvq163a184InaiQ88f46t7a5vz2mdUD0rtTulx7rqNJU0dQ6Okdu+njPP30swc9R7BzLyBMiTKayLikE7LRMTfRsQTapMWAztHxDa1ZYYoD838zQTrPiER8ZiIeC/lIY3VwJFdlHlJRMzpMOuJ1Wv9IaKl1WunB30mw6n1h0qq8TPfX/1YH1vxNspT8a+q/26q/Ripx3g8df88sAZ4RxVG604GNgU+X394R9L05eVpSevJzN/x8IdQRvMGyoMOn4mIIymXc5dReit3A55JeeDl7mr5j1PGcrwxIi6nhIp9KIHxy5SBu/thfjXINZR7Lp9GeShjC8q9dodk5ve6WM/pwI7VoNSLKWHzOZRxJ39LGTOx5TuUXsxzI+IyylA0yzJzouMoUtX50cDPIuJLlMGxX0e5BHx22yfbrImIM4ATKL+HKynng/2B/6u+2v2AEoCPqsJoawilT7QP0F3bzuKIOAr4JPCTiPgC5R7RfSnHxG2U4ZIkDQB7GiVNSPWU8HMovXNrKZ9iciTlUunvKA+G3Fxb/hzKQzZ3Uj5t5kDKU7HPA37Sx6ofROndfD/lSfFdKINjHwzsnJmdxors5BTga8Cu1Xr+hdLLeAqwV2a2LgWTmd+gPDCzBvhXSm/buydjZyhhdT/gm8DrKe2+nPKU+ts7LP9B4HjK5eG3AS+nDCr+0qp+66n247XAzyltdHL1NeqTzJl5drXOH1blj6Y8Mf9R4Pn1YXUkTW9Dw8Pjud9akiRJjyT2NEqSJKmRoVGSJEmNDI2SJElqZGiUJElSI0OjJEmSGhkaJUmS1MjQKEmSpEaGRkmSJDUyNEqSJKmRoVGSJEmN/h/b1tHn+7gphwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f15293e61d0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,10))\n",
    "sns.distplot(p_diffs, bins=10, kde=False, rug=True);\n",
    "\n",
    "plt.xlabel('Mean Distribution', fontsize=20)\n",
    "plt.title('Distributions of 10,000 samples at size of 145,300', fontsize=24)\n",
    "plt.ylabel('Number of Samples', fontsize=20)\n",
    "plt.xticks(fontsize=12)\n",
    "plt.yticks(fontsize=12);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = np.array(p_diffs)\n",
    "null_vals = np.random.normal(0, p_diffs.std(), p_diffs.size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXkAAAD1CAYAAAC1BoUqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEvxJREFUeJzt3X+MXWWdx/H3MKPs1lam6IC1bVJ0y3cFoviDSkKWoCwIyAImKwtmpQIRswF/ZNm4gG4wIJtuFLSJLrsIXUsWhUZRGrdrrWxcYiLQlUXlx353KzYwtAJrKdI0KSnM/nGfYS/jnc7tnXtn5j59v5LJnPvc55zzPdPTzz3znOfeGRgbG0OSVKeDZrsASVLvGPKSVDFDXpIqZshLUsUMeUmqmCEvSRUbmu0CJnrmmednZU7n/PkHs2vXntnYdUf6rV7orOZDzjkDgOe+u6EXJe1Tv/2Mrbe35nq9IyMLBlq1eyVfDA0NznYJ+6Xf6oX+q9l6e8t6Z4YhL0kVM+QlqWKGvCRVzJCXpIoZ8pJUMUNekipmyEtSxQx5SarYnHvHqzSV466/Z1b2u/nyE2dlv9J0eCUvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoZ8pJUMT+FUmrTbH36JfgJmOqcV/KSVDFDXpIqZshLUsWmHJOPiKXArcAbgJeAmzJzdUR8Dvgo8EzpelVmbijrXAlcDLwIfCIzN5b204DVwCBwc2au6u7hSJKatXPjdS9weWY+EBELgJ9GxKby3Jcy84vNnSPiKOA84GjgjcAPI+LI8vRXgVOAUWBzRKzPzEe6cSCSpN81Zchn5nZge1l+PiIeBRbvY5Wzgdszcw/wq4jYAqwoz23JzMcAIuL20teQl6Qe2a8x+YhYBrwduK80XRYRP4+INRGxsLQtBp5oWm20tE3WLknqkbbnyUfEfODbwKcy87cRcSNwLTBWvl8PXAQMtFh9jNYvKGMTG+bPP5ihocF2y+qawcGDGB6eN+P77VS/1Qud1Tw41Dht+u1Yu62d4++3c8J6Z0ZbIR8Rr6IR8Ldl5p0AmflU0/NfA75XHo4CS5tWXwJsK8uTtb9s16497dbeVcPD89i5c/es7LsT/VYvdFbzIXtfAuC5PjvWbmvn59Zv54T1dtfIyIKW7e3MrhkAbgEezcwbmtoXlfF6gA8AD5Xl9cA3IuIGGjdelwP307jCXx4RRwBP0rg5+6GOjkaS1JZ2ruRPAD4M/CIiHixtVwHnR8SxNIZctgIfA8jMhyNiHY0bqnuBSzPzRYCIuAzYSGMK5ZrMfLiLxyJJmqCd2TU/pvU4+4Z9rHMdcF2L9g37Wk+S1F2+41WSKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoZ8pJUMUNekipmyEtSxQx5SaqYIS9JFTPkJalihrwkVcyQl6SKGfKSVDFDXpIqZshLUsUMeUmqmCEvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVG5qqQ0QsBW4F3gC8BNyUmasj4lDgDmAZsBU4NzOfjYgBYDVwBrAb+EhmPlC2tRL4bNn05zNzbXcPR5LUrJ0r+b3A5Zn5FuB44NKIOAq4Arg7M5cDd5fHAKcDy8vXJcCNAOVF4Wrg3cAK4OqIWNjFY5EkTTBlyGfm9vEr8cx8HngUWAycDYxfia8FzinLZwO3ZuZYZt4LDEfEIuB9wKbM3JGZzwKbgNO6ejSSpFfYrzH5iFgGvB24Dzg8M7dD44UAOKx0Www80bTaaGmbrF2S1CNTjsmPi4j5wLeBT2XmbyNisq4DLdrG9tH+CvPnH8zQ0GC7ZXXN4OBBDA/Pm/H9dqrf6oXOah4calyH9Nuxdls7x99v54T1zoy2Qj4iXkUj4G/LzDtL81MRsSgzt5fhmKdL+yiwtGn1JcC20n7ShPYfTdzXrl179qf+rhkensfOnbtnZd+d6Ld6obOaD9n7EgDP9dmxdls7P7d+Oyest7tGRha0bJ9yuKbMlrkFeDQzb2h6aj2wsiyvBO5qar8gIgYi4njguTKcsxE4NSIWlhuup5Y2SVKPtHMlfwLwYeAXEfFgabsKWAWsi4iLgceBD5bnNtCYPrmFxhTKCwEyc0dEXAtsLv2uycwdXTkKSVJLU4Z8Zv6Y1uPpACe36D8GXDrJttYAa/anQElS53zHqyRVzJCXpIoZ8pJUMUNekipmyEtSxQx5SaqYIS9JFTPkJalihrwkVcyQl6SKGfKSVDFDXpIqZshLUsXa/stQUrPjrr9nRvZz+xM7AThvhvYn1cYreUmqmCEvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoZ8pJUsSk/ajgi1gBnAk9n5jGl7XPAR4FnSrerMnNDee5K4GLgReATmbmxtJ8GrAYGgZszc1V3D0WSNFE7nyf/deArwK0T2r+UmV9sboiIo4DzgKOBNwI/jIgjy9NfBU4BRoHNEbE+Mx+ZRu2SpClMOVyTmfcAO9rc3tnA7Zm5JzN/BWwBVpSvLZn5WGa+ANxe+kqSemg6Y/KXRcTPI2JNRCwsbYuBJ5r6jJa2ydolST3U6Z//uxG4Fhgr368HLgIGWvQdo/WLyVirDc+ffzBDQ4MdltW5wcGDGB6eN+P77VS/1avpaeffut/OCeudGR2FfGY+Nb4cEV8DvlcejgJLm7ouAbaV5cnaX2HXrj2dlDRtw8Pz2Llz96zsuxP9Vq+mp51/6347J6y3u0ZGFrRs7yjkI2JRZm4vDz8APFSW1wPfiIgbaNx4XQ7cT+MKf3lEHAE8SePm7Ic62bckqX3tTKH8JnAS8PqIGAWuBk6KiGNpDLlsBT4GkJkPR8Q64BFgL3BpZr5YtnMZsJHGFMo1mflw149GkvQKU4Z8Zp7fovmWffS/DriuRfsGYMN+VScJgOOuv2dW9rv58hNnZb/qHt/xKkkVM+QlqWKGvCRVzJCXpIoZ8pJUMUNekipmyEtSxQx5SaqYIS9JFTPkJalihrwkVcyQl6SKGfKSVDFDXpIqZshLUsUMeUmqmCEvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoNTdUhItYAZwJPZ+Yxpe1Q4A5gGbAVODczn42IAWA1cAawG/hIZj5Q1lkJfLZs9vOZuba7hyJJmqidK/mvA6dNaLsCuDszlwN3l8cApwPLy9clwI3w8ovC1cC7gRXA1RGxcLrFS5L2bcqQz8x7gB0Tms8Gxq/E1wLnNLXfmpljmXkvMBwRi4D3AZsyc0dmPgts4ndfOCRJXdbpmPzhmbkdoHw/rLQvBp5o6jda2iZrlyT10JRj8vtpoEXb2D7af8f8+QczNDTY1aLaMTh4EMPD82Z8v53qt3rVn3p5jvXbOdxv9Y7rNOSfiohFmbm9DMc8XdpHgaVN/ZYA20r7SRPaf9Rqw7t27emwpOkZHp7Hzp27Z2Xfnei3etWfenmO9ds5PNfrHRlZ0LK90+Ga9cDKsrwSuKup/YKIGIiI44HnynDORuDUiFhYbrieWtokST3UzhTKb9K4Cn99RIzSmCWzClgXERcDjwMfLN030Jg+uYXGFMoLATJzR0RcC2wu/a7JzIk3cyVJXTZlyGfm+ZM8dXKLvmPApZNsZw2wZr+qkyRNi+94laSKGfKSVDFDXpIqZshLUsUMeUmqmCEvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoZ8pJUMUNekio25d941dx23PX3zHYJkuYwr+QlqWKGvCRVzJCXpIoZ8pJUMUNekipmyEtSxQx5SaqYIS9JFTPkJali03rHa0RsBZ4HXgT2Zua7IuJQ4A5gGbAVODczn42IAWA1cAawG/hIZj4wnf1LkvatG1fy78nMYzPzXeXxFcDdmbkcuLs8BjgdWF6+LgFu7MK+JUn70IvhmrOBtWV5LXBOU/utmTmWmfcCwxGxqAf7lyQV0/2AsjHgBxExBvxjZt4EHJ6Z2wEyc3tEHFb6LgaeaFp3tLRtb97g/PkHMzQ0OM2y9t/g4EEMD8+b8f12qt/qVX/q5TnWb+dwv9U7brohf0JmbitBviki/msffQdatI1NbNi1a880S+rM8PA8du7cPSv77kS/1av+1MtzrN/O4ble78jIgpbt0xquycxt5fvTwHeAFcBT48Mw5fvTpfsosLRp9SXAtunsX5K0bx2HfES8JiIWjC8DpwIPAeuBlaXbSuCusrweuCAiBiLieOC58WEdSVJvTGe45nDgOxExvp1vZOb3I2IzsC4iLgYeBz5Y+m+gMX1yC40plBdOY9+SZsBs/lGazZefOGv7rknHIZ+ZjwFva9H+G+DkFu1jwKWd7k+StP98x6skVcyQl6SKGfKSVDFDXpIqZshLUsUMeUmqmCEvSRUz5CWpYoa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kVM+QlqWKGvCRVzJCXpIoZ8pJUMUNekirW8R/y1ivN5l+1l6TJeCUvSRUz5CWpYoa8JFXMMXlJc9Js3efafPmJs7LfXvFKXpIqZshLUsUMeUmq2IyPyUfEacBqYBC4OTNXzXQNknSgmNGQj4hB4KvAKcAosDki1mfmI93Yvm9IkqRXmunhmhXAlsx8LDNfAG4Hzp7hGiTpgDHTwzWLgSeaHo8C727uMDKyYKDTjW9d9f5OV9VcVf5Nt85uFRIAIyMLZruE/TbTV/KtAnxshmuQpAPGTIf8KLC06fESYNsM1yBJB4yZHq7ZDCyPiCOAJ4HzgA/NcA2SdMAYGBub2dGSiDgD+DKNKZRrMvO6Hu/vUOAOYBmNod1zM/PZFv1WAp8tDz+fmWtL+zuBrwO/D2wAPpmZY+W5jwOXAXuBf8nMT8/lesvzfwV8ARjJzP+dq/VGxBeAPwFeAH4JXJiZO6dR5z6n7kbEwcCtwDuB3wB/lplby3NXAhcDLwKfyMyN7WxzOrpdb0QsLf3fALwE3JSZq+dqvU3rDQL/ATyZmWfO5XojYhi4GTiGxjD0RZn5k27V3KkZfzNUZm7IzCMz8829DvjiCuDuzFwO3F0ev0IJqqtp3AReAVwdEQvL0zcClwDLy9dpZZ330JgZ9NbMPBr44lyut6y3lMb01ce7VGsv690EHJOZbwX+G7iy0wKbpu6eDhwFnB8RR03odjHwbGb+AfAl4O/KukfR+I3z6FLb30fEYJvbnDP10rgQuTwz3wIcD1w6x+sd90ng0W7UOQP1rga+n5l/CLyt23V36kB4x+vZwNqyvBY4p0Wf9wGbMnNHuQrdBJwWEYuA12bmT8rV8K1N6/8FsCoz9wBk5tNzvF5onKyfprs3u3tSb2b+IDP3lvXvpXH/plPtTN1tPo5vASdHxEBpvz0z92Tmr4AtZXu9nA7c9Xozc3tmPgCQmc/TCKDFc7VegIhYAryfxtVxN3W93oh4LXAicAtAZr4wnd88u+lACPnDM3M7QPl+WIs+raZ2Li5foy3aAY4E/igi7ouIf4+I4+ZyvRFxFo1feX/WpTp7Wu8EFwH/Oo0aJ9t/yz7lxeU54HVT1D7VNudSvS+LiGXA24H75ni9X6ZxUfJSl+rsZb1vAp4B/iki/jMibo6I13S57o5U8VHDEfFDGmONE32mzU1MNrVzX1M+h4CFNH71PQ5YFxFvah7/nsxM1xsR88q2T21z+68wSz/f8X1/hsZQw21t7mt/9t9On8naW10gdes3pF7UC0BEzAe+DXwqM3/bcYXt1dJOn8nO2TOBpzPzpxFx0jTrm6gXP98h4B3AxzPzvohYTWPo8m+mU2g3VBHymfnHkz0XEU9FxKLM3F6GB1oNq4wCJzU9XgL8qLQvmdC+rWmdO0uo3x8RLwGvp/FqPtfqfTNwBPCziBhvfyAiVmTmr+dgvePbXgmcCZzczovnPrQzdXe8z2hEDAGHADumWLdX04F7Um9EvIpGwN+WmXd2qdZe1XsWcFaZqPF7wGsj4p8z88/naL2jwGhmjv929C1a3J+aDQfCcM16YGVZXgnc1aLPRuDUiFhYbgieCmwsww/PR8TxZTzugqb1vwu8FyAijgReDUx7tkov6s3MX2TmYZm5LDOX0Tgh39FOwM9GvfDy7Ie/Bs7KzN3TrPHlqbsR8WoaN87W7+M4/hT4t/LCsh44LyIOLlN/lwP3t7nNOVNv+fneAjyamTd0qc6e1ZuZV2bmknK+nlf6dyPge1Xvr4EnolxFAScDXflMruk6EEJ+FXBKRPwPjZklqwAi4l0RcTNAZu4ArqXxj78ZuKa0QeMG6800brD8kv8fG14DvCkiHqJx42blNK82e11vr/Sq3q8AC4BNEfFgRPxDpwWWMdXLaLzYPAqsy8yHI+Kacq8CGgH4uojYAvwl5SosMx8G1tH4D/t94NLMfHGybXZaY6/rBU4APgy8t/w8HyxXyXO13p7pYb0fB26LiJ8DxwJ/28vjaNeMz5OXJM2cA+FKXpIOWIa8JFXMkJekihnyklQxQ16SKmbIS1LFDHlJqpghL0kV+z/R7WItFqpS3wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f152024bb70>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(null_vals);\n",
    "plt.axvline(x=obs_diff, color='red');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## p-value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.76852314768523144"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate the p-value \n",
    "# These are all the null values that are more extreme than our statistic in favor of our alternative\n",
    "\n",
    "(null_vals > obs_diff).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "j. What proportion of the **p_diffs** are greater than the actual difference observed in **ab_data.csv**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual difference in new & old conversion means: -0.0015782389853555567\n"
     ]
    }
   ],
   "source": [
    "actual_new_conv = df2[df2.landing_page == 'new_page']['converted'].mean()\n",
    "actual_old_conv = df2[df2.landing_page == 'old_page']['converted'].mean()\n",
    "\n",
    "actual_diff = actual_new_conv - actual_old_conv\n",
    "\n",
    "print(f\"Actual difference in new & old conversion means: {actual_diff}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "diff_prop = np.greater(p_diffs,actual_diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80611938806119388"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Proportion of p_diffs that are greater than the actual difference observed in ab_data\n",
    "diff_prop.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k. Please explain using the vocabulary you've learned in this course what you just computed in part **j.**  What is this value called in scientific studies?  What does this value mean in terms of whether or not there is a difference between the new and old pages?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilizing the ab_data, we set out to determine whether the null hypothesis, **uP_new - uP_old <= 0**, the mean conversion rate of the new webpage minus the mean conversation rate of the old webpage is less than or equal to 0. In other words, that the old and new webpages produce the same (or near same) conversation rates.\n",
    "\n",
    "The p-value calculated under heading p-value, indicates that **94%** of all the null values are more extreme than our statistic in favor of the alternative.\n",
    "\n",
    "Therefore, we **failed to reject the null hypothesis** which essentially means there is not enough evidence to say that it is false.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l. Fill in the below to calculate the number of conversions for each page, as well as the number of individuals who received each page. Let `n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### z-test attempt as well as setup for stats.proportions_ztest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.\n",
      "  from pandas.core import datetools\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "145274 0.1203863045 0.32541384592\n",
      "145310 0.118808065515 0.323563640675\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "old_data = df2[df2.landing_page == 'old_page']['converted']\n",
    "mu_convert_old = df2[df2.landing_page == 'old_page']['converted'].mean()\n",
    "std_convert_old = df2[df2.landing_page == 'old_page']['converted'].std()\n",
    "\n",
    "new_data = df2[df2.landing_page == 'new_page']['converted']\n",
    "mu_convert_new = df2[df2.landing_page == 'new_page']['converted'].mean()\n",
    "std_convert_new = df2[df2.landing_page == 'new_page']['converted'].std()\n",
    "\n",
    "n_old = len(df2[df2.group == 'control'])\n",
    "n_new = len(df2[df2.group == 'treatment'])\n",
    "\n",
    "print(n_old,mu_convert_old,std_convert_old)\n",
    "print(n_new,mu_convert_new,std_convert_new)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.3109235634981506, 0.18988358901317048)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.stats.ztest(old_data,new_data,alternative='two-sided')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mu_convert_old"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mu_convert_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "m. Now use `stats.proportions_ztest` to compute your test statistic and p-value.  [Here](https://docs.w3cub.com/statsmodels/generated/statsmodels.stats.proportion.proportions_ztest/) is a helpful link on using the built in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.859354929150913, 0.96851157707008961)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sm.stats.proportions_ztest(mu_convert_new * n_new, n_new, mu_convert_old, alternative='larger')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n. What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The z-score reveals that it is **-1.9 standard deviations** away from the mean. The p-value is even higher than our manual hypothesis testing and suggests that under the null hypothesis, the difference in the new & old conversion rates are **0, and is 97%** in favor of our null  **\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "`1.` In this final part, you will see that the result you achieved in the A/B test in Part II above can also be achieved by performing regression.<br><br> \n",
    "\n",
    "a. Since each row is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Logistic Regression**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b. The goal is to use **statsmodels** to fit the regression model you specified in part **a.** to see if there is a significant difference in conversion based on which page a customer receives. However, you first need to create in df2 a column for the intercept, and create a dummy variable column for which page each user received.  Add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>treatment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "\n",
       "   intercept  control  treatment  \n",
       "0          1        1          0  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create numerical dummy columns for the group categorical column\n",
    "df2['intercept'] = 1\n",
    "df2[['control','treatment']] = pd.get_dummies(df2['group'])\n",
    "df2.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>treatment</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "5   936923  2017-01-10 15:20:49.083499    control     old_page          0   \n",
       "6   679687  2017-01-19 03:26:46.940749  treatment     new_page          1   \n",
       "7   719014  2017-01-17 01:48:29.539573    control     old_page          0   \n",
       "8   817355  2017-01-04 17:58:08.979471  treatment     new_page          1   \n",
       "9   839785  2017-01-15 18:11:06.610965  treatment     new_page          1   \n",
       "\n",
       "   intercept  control  treatment  new_page  old_page  \n",
       "0          1        1          0         0         1  \n",
       "1          1        1          0         0         1  \n",
       "2          1        0          1         1         0  \n",
       "3          1        0          1         1         0  \n",
       "4          1        1          0         0         1  \n",
       "5          1        1          0         0         1  \n",
       "6          1        0          1         1         0  \n",
       "7          1        1          0         0         1  \n",
       "8          1        0          1         1         0  \n",
       "9          1        0          1         1         0  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create numerical dummy columns for the landing_page categorical column\n",
    "df2[['new_page','old_page']] = pd.get_dummies(df2['landing_page'])\n",
    "df2.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>control</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>old_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  control  ab_page  new_page  old_page  \n",
       "0          1        1        0         0         1  \n",
       "1          1        1        0         0         1  \n",
       "2          1        0        1         1         0  \n",
       "3          1        0        1         1         0  \n",
       "4          1        1        0         0         1  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename the treatment column to ab_page\n",
    "df2 = df2.rename(columns={'treatment':'ab_page'})\n",
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove control column. It's values are represented by a 0 in the ab_page column\n",
    "df2.drop(columns=['control'],inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "\n",
       "   intercept  ab_page  new_page  \n",
       "0          1        0         0  \n",
       "1          1        0         0  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove old_page column. It's values are represented by a 0 in the new_page column\n",
    "df2.drop(columns=['old_page'], inplace=True)\n",
    "df2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "c. Use **statsmodels** to instantiate your regression model on the two columns you created in part b., then fit the model using the two columns you created in part **b.** to predict whether or not an individual converts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "log_mod = sm.Logit(df2['converted'], df2[['intercept','ab_page']])\n",
    "results = log_mod.fit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "d. Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2020-10-14 16:34</td>       <td>AIC:</td>        <td>212780.3502</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212801.5095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>1</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290582</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3109</td>  <td>0.1899</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2020-10-14 16:34 AIC:              212780.3502\n",
       "No. Observations:   290584           BIC:              212801.5095\n",
       "Df Model:           1                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290582           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "ab_page      -0.0150    0.0114    -1.3109  0.1899  -0.0374   0.0074\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "e. What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Logistic Regression model, the NULL & ALT hypothesis are:  \n",
    "H0: uP_new = uP_old  \n",
    "H1: uP_new != uP_old  \n",
    "  \n",
    "#### In Part II, the NULL & ALT hypothesis are:  \n",
    "H0: uP_new - uP_old <= 0  \n",
    "H1: uP_new - uP_old > 0  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The p-value associated with the **ab_page** is **0.19**. It differs from the null & alt hypothesis in Part II because, as hinted, the hypothesis for the Logistic Reg. model in says that the new page conversion will equal the old page conversion, which is a much more definitive statement, and leaves no room for small differences between the 2 conversions. Whereas, the Part II hypothesis give a much larger area for the NULL to be True. The number of values that satisfy the **uP_new - uP_old <= 0** is much great than **uP_new = uP_old**.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "f. Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a good idea to see what other factors can influence your models and the relationships between the variables. However, there are numerous problems that one may contend with when adding additional variables. **Linearity, Correlated Errors, Non-constant Variance and Normally Distributed Errors, Outliers/Leverage points, & Multi-collinearity** are some issues that may or may not arise with more complex models.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "g. Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in.\n",
    "\n",
    "Does it appear that country had an impact on conversion? Provide the statistical output as well as a written response to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read in countries.csv\n",
    "country_df = pd.read_csv(\"countries.csv\")\n",
    "country_df.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>CA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "5   936923  2017-01-10 15:20:49.083499    control     old_page          0   \n",
       "6   679687  2017-01-19 03:26:46.940749  treatment     new_page          1   \n",
       "7   719014  2017-01-17 01:48:29.539573    control     old_page          0   \n",
       "8   817355  2017-01-04 17:58:08.979471  treatment     new_page          1   \n",
       "9   839785  2017-01-15 18:11:06.610965  treatment     new_page          1   \n",
       "\n",
       "   intercept  ab_page  new_page country  \n",
       "0          1        0         0      US  \n",
       "1          1        0         0      US  \n",
       "2          1        1         1      US  \n",
       "3          1        1         1      US  \n",
       "4          1        0         0      US  \n",
       "5          1        0         0      US  \n",
       "6          1        1         1      CA  \n",
       "7          1        0         0      US  \n",
       "8          1        1         1      UK  \n",
       "9          1        1         1      CA  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# merge countries dataframe with df3 dataframe. Using new df3 dataframe to preserve the df2 dataframe\n",
    "df3 = pd.merge(df2, country_df, on='user_id')\n",
    "df3.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country  landing_page\n",
       "CA       new_page          7301\n",
       "         old_page          7198\n",
       "UK       old_page         36360\n",
       "         new_page         36106\n",
       "US       new_page        101903\n",
       "         old_page        101716\n",
       "Name: landing_page, dtype: int64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# see how the new_page & old_page counts are represented by country\n",
    "df3.groupby(df3['country'])['landing_page'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp    group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "\n",
       "   intercept  ab_page  new_page country  CA  UK  US  \n",
       "0          1        0         0      US   0   0   1  \n",
       "1          1        0         0      US   0   0   1  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create numerical dummy columns for categorical country column\n",
    "df3[['CA','UK','US']] = pd.get_dummies(df3['country'])\n",
    "df3.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>CA</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>UK</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>CA</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "5   936923  2017-01-10 15:20:49.083499    control     old_page          0   \n",
       "6   679687  2017-01-19 03:26:46.940749  treatment     new_page          1   \n",
       "7   719014  2017-01-17 01:48:29.539573    control     old_page          0   \n",
       "8   817355  2017-01-04 17:58:08.979471  treatment     new_page          1   \n",
       "9   839785  2017-01-15 18:11:06.610965  treatment     new_page          1   \n",
       "\n",
       "   intercept  ab_page  new_page country  CA  UK  US  \n",
       "0          1        0         0      US   0   0   1  \n",
       "1          1        0         0      US   0   0   1  \n",
       "2          1        1         1      US   0   0   1  \n",
       "3          1        1         1      US   0   0   1  \n",
       "4          1        0         0      US   0   0   1  \n",
       "5          1        0         0      US   0   0   1  \n",
       "6          1        1         1      CA   1   0   0  \n",
       "7          1        0         0      US   0   0   1  \n",
       "8          1        1         1      UK   0   1   0  \n",
       "9          1        1         1      CA   1   0   0  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# verify if dummy creation accurate\n",
    "df3.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366116\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2020-10-14 16:34</td>       <td>AIC:</td>        <td>212780.8333</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212812.5723</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>2</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290581</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9967</td>  <td>0.0068</td>  <td>-292.3145</td> <td>0.0000</td> <td>-2.0101</td> <td>-1.9833</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>CA</th>        <td>-0.0408</td>  <td>0.0269</td>   <td>-1.5178</td>  <td>0.1291</td> <td>-0.0935</td> <td>0.0119</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0099</td>   <td>0.0133</td>   <td>0.7458</td>   <td>0.4558</td> <td>-0.0161</td> <td>0.0360</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2020-10-14 16:34 AIC:              212780.8333\n",
       "No. Observations:   290584           BIC:              212812.5723\n",
       "Df Model:           2                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290581           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9967    0.0068  -292.3145  0.0000  -2.0101  -1.9833\n",
       "CA           -0.0408    0.0269    -1.5178  0.1291  -0.0935   0.0119\n",
       "UK            0.0099    0.0133     0.7458  0.4558  -0.0161   0.0360\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create logistic model utilizing the US as the intercept\n",
    "log_mod2 = sm.Logit(df3['converted'], df3[['intercept','CA','UK']])\n",
    "results = log_mod2.fit()\n",
    "\n",
    "# display the results\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, adding the country data to the model does not increase the likiness of a page conversion, with **p-values: 0.13, & 0.46**. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "h. Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there significant effects on conversion.  Create the necessary additional columns, and fit the new model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "# create logistic model utilizing Canada (CA) as the intercept\n",
    "log_mod3 = sm.Logit(df3['converted'], df3[['intercept','ab_page','US','UK']])\n",
    "results = log_mod3.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2020-10-14 16:34</td>       <td>AIC:</td>        <td>212781.1253</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212823.4439</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>3</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290580</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-2.0300</td>  <td>0.0266</td>  <td>-76.2488</td> <td>0.0000</td> <td>-2.0822</td> <td>-1.9778</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0149</td>  <td>0.0114</td>   <td>-1.3069</td> <td>0.1912</td> <td>-0.0374</td> <td>0.0075</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>0.0408</td>   <td>0.0269</td>   <td>1.5161</td>  <td>0.1295</td> <td>-0.0119</td> <td>0.0934</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0506</td>   <td>0.0284</td>   <td>1.7835</td>  <td>0.0745</td> <td>-0.0050</td> <td>0.1063</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2020-10-14 16:34 AIC:              212781.1253\n",
       "No. Observations:   290584           BIC:              212823.4439\n",
       "Df Model:           3                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290580           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept     -2.0300    0.0266  -76.2488  0.0000  -2.0822  -1.9778\n",
       "ab_page       -0.0149    0.0114   -1.3069  0.1912  -0.0374   0.0075\n",
       "US             0.0408    0.0269    1.5161  0.1295  -0.0119   0.0934\n",
       "UK             0.0506    0.0284    1.7835  0.0745  -0.0050   0.1063\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# display the results\n",
    "results.summary2()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After adding the ab_page variable, the results remain similar. The data, including old or new page, country of origin, CA, UK, & US, does not give us sufficient evidents, at least evidence that falls below alpha of 0.05, that the conversion rate of the new page is better than the conversion rate of the old page. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Influences associated with time on conversion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add additional time data to a new dataframe to keep df3 at current state\n",
    "df4 = df3.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page  new_page country  CA  UK  US  \n",
       "0          1        0         0      US   0   0   1  \n",
       "1          1        0         0      US   0   0   1  \n",
       "2          1        1         1      US   0   0   1  \n",
       "3          1        1         1      US   0   0   1  \n",
       "4          1        0         0      US   0   0   1  "
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id          int64\n",
       "timestamp       object\n",
       "group           object\n",
       "landing_page    object\n",
       "converted        int64\n",
       "intercept        int64\n",
       "ab_page          uint8\n",
       "new_page         uint8\n",
       "country         object\n",
       "CA               uint8\n",
       "UK               uint8\n",
       "US               uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check datatypes, notice the timestamp column is an object, not a datetime type\n",
    "df4.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert datatype for timestamp column to datetime\n",
    "df4['timestamp'] = pd.to_datetime(df4['timestamp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id                  int64\n",
       "timestamp       datetime64[ns]\n",
       "group                   object\n",
       "landing_page            object\n",
       "converted                int64\n",
       "intercept                int64\n",
       "ab_page                  uint8\n",
       "new_page                 uint8\n",
       "country                 object\n",
       "CA                       uint8\n",
       "UK                       uint8\n",
       "US                       uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Jan  2 13:42:05 2017\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "earliest = df4.timestamp.min()\n",
    "latest = df4.timestamp.max()\n",
    "print(earliest.ctime())\n",
    "print(latest.dayofweek)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Mon', 'Jan', '2', '13:42:05', '2017']"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split_earl = earliest.ctime().split()\n",
    "split_earl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "print(earliest.dayofweek)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  timestamp    group landing_page  converted  \\\n",
       "0   851104 2017-01-21 22:11:48.556739  control     old_page          0   \n",
       "1   804228 2017-01-12 08:01:45.159739  control     old_page          0   \n",
       "\n",
       "   intercept  ab_page  new_page country  CA  UK  US  \n",
       "0          1        0         0      US   0   0   1  \n",
       "1          1        0         0      US   0   0   1  "
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to extract day of week from timestamp and categorize new column with \n",
    "# appropriate designation, weekend or weekday\n",
    "\n",
    "def add_day(row):\n",
    "    timestamp_day = row.timestamp\n",
    "    return(timestamp_day.ctime().split()[0])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to extract day of week from timestamp and categorize new column with \n",
    "# appropriate designation, weekend or weekday\n",
    "\n",
    "def weekend(row):\n",
    "    timestamp_day = row.timestamp.dayofweek\n",
    "       \n",
    "    if timestamp_day == 6 or timestamp_day == 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to extract day of week from timestamp and categorize new column with \n",
    "# appropriate designation, weekend or weekday\n",
    "\n",
    "def weekday(row):\n",
    "    timestamp_day = row.timestamp.dayofweek\n",
    "    #if timestamp_day == 6 or timestamp_day == 0:   \n",
    "    if timestamp_day > 0 and timestamp_day < 6:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "## function to extract time of day from timestamp and categorize new column with\n",
    "## appropriate designation, daytime or nighttime\n",
    "\n",
    "def daytime(row):\n",
    "    view_time = row.timestamp.ctime().split()[3]\n",
    "    \n",
    "    if view_time > '06:30:00' and view_time < '19:00:00':\n",
    "        return 1\n",
    "    else:\n",
    "        return 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run weekend function on every row to categorize the timestamp as occurring\n",
    "# on the weekend \n",
    "\n",
    "df4['weekend'] = df4.apply(lambda row: weekend(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run weekday function on every row to categorize the timestamp as occurring\n",
    "# on the weekday \n",
    "\n",
    "df4['weekday'] = df4.apply(lambda row: weekday(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run daytime function on every row to categorize the timestamp as occurring\n",
    "# during the day or night\n",
    "\n",
    "df4['daytime'] = df4.apply(lambda row: daytime(row), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>new_page</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "      <th>weekend</th>\n",
       "      <th>weekday</th>\n",
       "      <th>daytime</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  timestamp      group landing_page  converted  \\\n",
       "0   851104 2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228 2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590 2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541 2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975 2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page  new_page country  CA  UK  US  weekend  weekday  daytime  \n",
       "0          1        0         0      US   0   0   1        0        1        0  \n",
       "1          1        0         0      US   0   0   1        0        1        1  \n",
       "2          1        1         1      US   0   0   1        0        1        1  \n",
       "3          1        1         1      US   0   0   1        1        0        1  \n",
       "4          1        0         0      US   0   0   1        0        1        0  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df4.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "log_mod4 = sm.Logit(df4['converted'], df4[['intercept','ab_page','weekday']])\n",
    "results = log_mod4.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2020-10-14 16:54</td>       <td>AIC:</td>        <td>212782.2460</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212813.9850</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>2</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290581</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9859</td>  <td>0.0120</td>  <td>-165.9746</td> <td>0.0000</td> <td>-2.0094</td> <td>-1.9625</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3103</td>  <td>0.1901</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>weekday</th>   <td>-0.0040</td>  <td>0.0125</td>   <td>-0.3229</td>  <td>0.7468</td> <td>-0.0286</td> <td>0.0205</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2020-10-14 16:54 AIC:              212782.2460\n",
       "No. Observations:   290584           BIC:              212813.9850\n",
       "Df Model:           2                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290581           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9859    0.0120  -165.9746  0.0000  -2.0094  -1.9625\n",
       "ab_page      -0.0150    0.0114    -1.3103  0.1901  -0.0374   0.0074\n",
       "weekday      -0.0040    0.0125    -0.3229  0.7468  -0.0286   0.0205\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "log_mod5 = sm.Logit(df4['converted'], df4[['intercept','ab_page','daytime']])\n",
    "results = log_mod5.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2020-10-14 16:55</td>       <td>AIC:</td>        <td>212776.9638</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212808.7028</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>2</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290581</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-2.0027</td>  <td>0.0101</td>  <td>-199.0636</td> <td>0.0000</td> <td>-2.0224</td> <td>-1.9830</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3160</td>  <td>0.1882</td> <td>-0.0375</td> <td>0.0074</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>daytime</th>   <td>0.0266</td>   <td>0.0115</td>   <td>2.3205</td>   <td>0.0203</td> <td>0.0041</td>  <td>0.0490</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2020-10-14 16:55 AIC:              212776.9638\n",
       "No. Observations:   290584           BIC:              212808.7028\n",
       "Df Model:           2                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290581           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -2.0027    0.0101  -199.0636  0.0000  -2.0224  -1.9830\n",
       "ab_page      -0.0150    0.0114    -1.3160  0.1882  -0.0375   0.0074\n",
       "daytime       0.0266    0.0115     2.3205  0.0203   0.0041   0.0490\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilizing these methods:\n",
    "\n",
    "1. Probability\n",
    "2. A/B Test\n",
    "3. zTest\n",
    "4. Logistic Regression\n",
    "\n",
    "it was unaminously calculated that the new_page design **DID NOT** provide enough statistical & practical evidence to confirm the likelness that conversions would increase consistently under the Type II error rate of 0.05."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='conclusions'></a>\n",
    "## Finishing Up\n",
    "\n",
    "> Congratulations!  You have reached the end of the A/B Test Results project!  You should be very proud of all you have accomplished!\n",
    "\n",
    "> **Tip**: Once you are satisfied with your work here, check over your report to make sure that it is satisfies all the areas of the rubric (found on the project submission page at the end of the lesson). You should also probably remove all of the \"Tips\" like this one so that the presentation is as polished as possible.\n",
    "\n",
    "\n",
    "## Directions to Submit\n",
    "\n",
    "> Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).\n",
    "\n",
    "> Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.\n",
    "\n",
    "> Once you've done this, you can submit your project by clicking on the \"Submit Project\" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "#from subprocess import call\n",
    "#call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}