IS6400-2122B-CityU-IS-Week-6_to_8-Homework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "economic-contrary",
   "metadata": {},
   "source": [
    "IS6400 21/22 Semester B\n",
    "\n",
    "- Homework exercises of Week 6 to Week 8\n",
    "\n",
    "Zhiya Zuo (zhiyazuo@cityu.edu.hk)\n",
    "\n",
    "Department of Information Systems\n",
    "\n",
    "City University of Hong Kong"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aquatic-honduras",
   "metadata": {},
   "source": [
    "__NOTE__: This is a notebook of homework exercises for materials from Week 6 to Week 8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pharmaceutical-batman",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "optional-singer",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:28.917570Z",
     "start_time": "2022-03-03T15:08:28.700110Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surface-patient",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:29.796708Z",
     "start_time": "2022-03-03T15:08:29.404490Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import metrics, linear_model, model_selection, preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "solved-glossary",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:30.039641Z",
     "start_time": "2022-03-03T15:08:29.798714Z"
    }
   },
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "# you need to run the following command if using colab\n",
    "# !pip install --upgrade statsmodels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "driven-alert",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:30.221571Z",
     "start_time": "2022-03-03T15:08:30.040752Z"
    }
   },
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "experienced-galaxy",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suburban-secret",
   "metadata": {},
   "source": [
    "# Prediction via Regression [40%]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "developed-replica",
   "metadata": {},
   "source": [
    "## Concepts [20%]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dependent-memorabilia",
   "metadata": {},
   "source": [
    "- Why do we say _ordinary least squared (OLS) regression_? HINT: recall what error linear regression is trying to minimize? __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7fa685d",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vertical-rebound",
   "metadata": {},
   "source": [
    "- What is overfitting? Describe in your own words __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2bf1583",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "significant-elder",
   "metadata": {},
   "source": [
    "- In a linear regression model, what are our solutions to overfitting? Please name two solutions with justifications. __[5%x2]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35311b2f",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "reserved-dietary",
   "metadata": {},
   "source": [
    "## Hands-on Applications [20%]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "broadband-warner",
   "metadata": {},
   "source": [
    "Read the dataset `cpu.csv` using `pandas.read_csv` to conduct the following analysis. Data source: https://raw.githubusercontent.com/zhiyzuo/IS6400-Regression/main/Data/cpu.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "silent-battlefield",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:33.494150Z",
     "start_time": "2022-03-03T15:08:33.046641Z"
    }
   },
   "outputs": [],
   "source": [
    "cpu_data = pd.read_csv('https://raw.githubusercontent.com/zhiyzuo/IS6400-Regression/main/Data/cpu.csv')\n",
    "# set the values from integer to floating point numbers\n",
    "cpu_data = cpu_data.astype(float)\n",
    "cpu_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2a23bd2",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:08:35.410856Z",
     "start_time": "2022-03-03T15:08:35.405458Z"
    }
   },
   "outputs": [],
   "source": [
    "X, y = cpu_data.loc[:, cpu_data.columns[:-1]].values, cpu_data['y'].values\n",
    "X.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "amazing-programmer",
   "metadata": {},
   "source": [
    "1. Spilt the dataset into training and testing randomly with 80% being training set. You need to print the number of data points in both the training and testing sets. __[4%]__\n",
    "    - HINT: use `numpy.random.choice` as shown in Week 6 slides or `sklearn.model_selection.train_test_split` following this [link](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae6e8ed9",
   "metadata": {},
   "source": [
    "Once you finish, please run the following code to standardize `X`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "142460cd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:12:12.889509Z",
     "start_time": "2022-03-03T15:12:12.789191Z"
    }
   },
   "outputs": [],
   "source": [
    "scaler = preprocessing.StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train)\n",
    "X_test = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "worldwide-surname",
   "metadata": {},
   "source": [
    "2. Train the regression model and do testing:\n",
    "    - Use a multiple linear regression to predict the target variable `y` __[2%]__ \n",
    "        - HINT: use the `.predict` function after `.fit`\n",
    "    - Print out the training error and testing error (mean squared error). __[2%]__\n",
    "        - HINT: use [metrics.mean_squared_errors](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.mean_squared_error.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cleared-piece",
   "metadata": {},
   "source": [
    "3. Let's now do the following modelling:\n",
    "    1. Apply both ridge regression and the linear regression models. You can set the hyperparemter in the ridge regression to 0.05. __[6%]__\n",
    "        - For ridge regression, do `linear_model.Ridge(alpha=0.05)`\n",
    "    2. Use 5-fold cross validation and show the average test error (prediction error on the held-out test set) between these two models. __[6%]__\n",
    "        - Use `model_selection.KFold(n_splits=5, shuffle=True)`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "copyrighted-lover",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ranking-british",
   "metadata": {},
   "source": [
    "# Time Series Forecasting [60%] (+ [5%] bonus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "accomplished-anniversary",
   "metadata": {},
   "source": [
    "## Concepts [25%]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb609c43",
   "metadata": {},
   "source": [
    "- Quantitative forecast is always better than qualitative forecasting. Yes or no? Why? __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "engaged-minute",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "chemical-atlantic",
   "metadata": {},
   "source": [
    "- What does autocorrelation capture for a time series dataset? __[2.5%]__ How about partial autocorrelation? __[2.5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48401256",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "suburban-despite",
   "metadata": {},
   "source": [
    "- What are the three components in a time series dataset? __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "975574fc",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "better-favorite",
   "metadata": {},
   "source": [
    "- What does it mean for a time series to be stationary? __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bb05a2",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seeing-amount",
   "metadata": {},
   "source": [
    "- What are the common approaches to data partioning for forecast performance evaluation? Provide two methods __[2.5%x2]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "358b378f",
   "metadata": {},
   "source": [
    "> [Your answer goes here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "strange-region",
   "metadata": {},
   "source": [
    "## Hands-on [35%] (+ [5%] bonus)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ultimate-listing",
   "metadata": {},
   "source": [
    "Read the data from this [lnik](https://raw.githubusercontent.com/robjhyndman/ETC3550Slides/fable/data/ausbeer.csv) using `pandas.read_csv` and conduct the following analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alike-brazilian",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:09:04.552164Z",
     "start_time": "2022-03-03T15:09:04.131004Z"
    }
   },
   "outputs": [],
   "source": [
    "url = 'https://raw.githubusercontent.com/robjhyndman/ETC3550Slides/fable/data/ausbeer.csv'\n",
    "beer_data = pd.read_csv(url)\n",
    "beer_data.Time = pd.to_datetime(beer_data.Time)\n",
    "beer_data.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9370bda1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:09:04.565264Z",
     "start_time": "2022-03-03T15:09:04.553985Z"
    }
   },
   "outputs": [],
   "source": [
    "beer_data = beer_data.set_index('Time')['Production'].resample('Q').mean()\n",
    "beer_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "informed-buyer",
   "metadata": {},
   "source": [
    "1. Visualize the time series data using line plot (x-axis is time and y is time series value). To reduce the number of `xticks`, use `ax.xaxis.set_major_locator(plt.MaxNLocator(5))` or follow the code in Week 7 Jupyter notebook. __[5%]__\n",
    "    - HINT: use `plt.subplots` and `beer_data.plot` functions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "approximate-tenant",
   "metadata": {},
   "source": [
    "2. Please answer the two questions:\n",
    "    1. Do you think it is stationary based on the figure above? Why? __[5%]__\n",
    "    1. Confirm with the KPSS test (you need to use `statsmodels` and check the p value of the returned statistics). HINT: read [link](https://www.statsmodels.org/devel/generated/statsmodels.tsa.stattools.kpss.html) on the `kpss` test. __[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a3ed34",
   "metadata": {},
   "source": [
    "3. Split the dataset into train and test sets. The training data should be 90% of the `beer_data` and the testing data should be 10%. Be aware about the temporal order. You should print the size of the training and testing sets.__[5%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sufficient-jordan",
   "metadata": {},
   "source": [
    "4. Conduct forecasting with Holt-Winter's method (i.e., exponential smoothing with trend and seasonality) using `sm.tsa.ETSModel` with additive trend and additive seasonality. Please:\n",
    "    - Add a damping factor to the trend component __[3%]__.\n",
    "    - Compare the training and testing error __[3%]__.\n",
    "    - HINT: You need to set `error`, `trend`,`damped_trend`, `seasonal`, and `seasonal_periods` (this is a quarterly dataset, what period should we set?)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d9e4fb3",
   "metadata": {},
   "source": [
    "5. Conduct forecasting similar as above but use an ARIMA model. __[5%]__\n",
    "    - How do you pick $p$, $d$, and $q$? Look at the (partial) autocorrelation diagrams!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2ef6efb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:09:11.061625Z",
     "start_time": "2022-03-03T15:09:10.851349Z"
    }
   },
   "outputs": [],
   "source": [
    "# the shaded region is the 95% CI\n",
    "fig, (ax_acf, ax_pacf) = plt.subplots(figsize=[15, 4.5], ncols=2, sharey=True)\n",
    "_ = sm.graphics.tsa.plot_acf(beer_train, lags=15, ax=ax_acf, zero=False)\n",
    "_ = sm.graphics.tsa.plot_pacf(beer_train, lags=15, ax=ax_pacf, zero=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9db4bd5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-03-03T15:09:11.619865Z",
     "start_time": "2022-03-03T15:09:11.404618Z"
    }
   },
   "outputs": [],
   "source": [
    "# the shaded region is the 95% CI\n",
    "beer_train_diff = sm.tsa.statespace.tools.diff(beer_train, k_diff=1, k_seasonal_diff=0)\n",
    "fig, (ax_acf, ax_pacf) = plt.subplots(figsize=[15, 4.5], ncols=2, sharey=True)\n",
    "_ = sm.graphics.tsa.plot_acf(beer_train_diff, lags=15, ax=ax_acf, zero=False)\n",
    "_ = sm.graphics.tsa.plot_pacf(beer_train_diff, lags=15, ax=ax_pacf, zero=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controlled-discount",
   "metadata": {},
   "source": [
    "6. Compare the test error between ARIMA and Holt-Winter's using the mean squared error. __[4%]__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "575c8abc",
   "metadata": {},
   "source": [
    "7. Bonus: visualize three time series data __[5%]__:\n",
    "    - The true test set \n",
    "    - The predicted values in the test set by ARIMA\n",
    "    - The predicted values in the test set by Winter's method.\n",
    "    - You can use the code from Week 7 lecture notebooks or exercise notebooks directly with some minor edits."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "425.594px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}