Build Comment Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build Comment Classifier\n",
    "\n",
    "This notebook is to build the Stackoverflow comment classifier and output:\n",
    "- `sentence_scorer.pkl` -> `SentenceScorer` to score a given list of processed comment text\n",
    "- `comment_classifier.pkl` -> `BalancedRandomForestClassifier` Classifier model to predict whether comment is useful or not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and definitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from statistics import mean\n",
    "\n",
    "from comment_classifier.sentence_scorer import SentenceScorer\n",
    "from imblearn.ensemble import BalancedRandomForestClassifier\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer, TfidfVectorizer\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from sklearn.metrics import f1_score,accuracy_score,precision_score,recall_score,roc_auc_score,plot_confusion_matrix\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "comments_model_input_df = pd.read_csv('data/classifier/comments_model_input.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description\n",
    "\n",
    "Pre-processing comment (`comment_classifier/utils`):\n",
    "- Replace URLs, mentions and code\n",
    "- Remove stopwords\n",
    "- Remove special characters, except '?'\n",
    "- Tokenize and apply stemming\n",
    "\n",
    "\n",
    "\n",
    "Two-level Random forest classifier\n",
    "1. **SentenceScorer**: Tf-idf of each word in sentence, generating a text score with the classifier probability prediction\n",
    "2. **Comment classifier**: Uses the Sentence Score (first classifier), Sentence length and NLTK Text Sentiment to predict useful/not useful\n",
    "    - Note: Using 3 main features from the model as described in page 20 of \"Are comments on Stack Overflow well organized for easy retrieval by developers?\" paper"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First level: SentenceScorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object BaseShuffleSplit.split at 0x7f9a6083b190>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_data_df = comments_model_input_df[comments_model_input_df['CommentTextProc'].notnull()]\n",
    "\n",
    "sss_half = StratifiedShuffleSplit(n_splits=1, test_size=0.5)\n",
    "X = filtered_data_df[['CommentTextProc','CommentTextLen','Sentiment']]\n",
    "y = np.array(filtered_data_df['IsUsefulSum'])\n",
    "\n",
    "sss_half.split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F1 Score for Balanced Random Forest Classifier is  0.8078431372549019\n",
      "F1 Score for Balanced Random Forest Classifier is  0.7905138339920948\n",
      "F1 Score for Balanced Random Forest Classifier is  0.8314606741573034\n",
      "F1 Score for Balanced Random Forest Classifier is  0.8313725490196079\n",
      "F1 Score for Balanced Random Forest Classifier is  0.7968127490039841\n",
      "\n",
      "\n",
      "F1 Score: 0.8314606741573034\n",
      "Accuracy: 0.7540983606557377\n",
      "Precision: 0.9098360655737705\n",
      "Recall: 0.7655172413793103\n",
      "AUC: 0.738021778584392\n"
     ]
    }
   ],
   "source": [
    "filtered_data_df = comments_model_input_df[comments_model_input_df['CommentTextProc'].notnull()]\n",
    "sss = StratifiedShuffleSplit(n_splits=5, test_size=0.3)\n",
    "sss_half = StratifiedShuffleSplit(n_splits=1, test_size=0.5)\n",
    "\n",
    "X_base = filtered_data_df[['CommentTextProc','CommentTextLen','Sentiment']]\n",
    "y_base = np.array(filtered_data_df['IsUsefulSum'])\n",
    "\n",
    "current_f1_score = 0\n",
    "for first_half_index, second_half_index in sss_half.split(X, y):\n",
    "    X, y = X_base.iloc[first_half_index], y[first_half_index]\n",
    "    for train_index, test_index in StratifiedShuffleSplit(n_splits=5, test_size=0.1).split(X, y):\n",
    "        X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "        y_train, y_test = y[train_index], y[test_index]\n",
    "\n",
    "        tf_idf = TfidfVectorizer()\n",
    "        X_train_tf = tf_idf.fit_transform(X_train['CommentTextProc'])\n",
    "        X_test_tf = tf_idf.transform(X_test['CommentTextProc'])\n",
    "\n",
    "        brfc = BalancedRandomForestClassifier().fit(X_train_tf, y_train)\n",
    "        scored_f1_score = f1_score(y_test,brfc.predict(X_test_tf))\n",
    "        print(\"F1 Score for Balanced Random Forest Classifier is \", scored_f1_score)\n",
    "\n",
    "        if scored_f1_score > current_f1_score:\n",
    "            current_tf_idf = tf_idf\n",
    "            current_f1_score = scored_f1_score\n",
    "            current_model = brfc\n",
    "            X_two, y_two = X_base.iloc[second_half_index], y_base[second_half_index]\n",
    "            current_y_test = y_test\n",
    "            current_X_train_tf = X_train_tf\n",
    "            current_X_test_tf = X_test_tf\n",
    "            current_y_train = y_test\n",
    "        \n",
    "y_predict = current_model.predict(current_X_test_tf)\n",
    "print(f\"\\n\\nF1 Score: {current_f1_score}\")\n",
    "print(f\"Accuracy: {accuracy_score(current_y_test, y_predict)}\")\n",
    "print(f\"Precision: {precision_score(current_y_test, y_predict)}\")\n",
    "print(f\"Recall: {recall_score(current_y_test, y_predict)}\")\n",
    "print(f\"AUC: {roc_auc_score(current_y_test, y_predict)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function plot_confusion_matrix is deprecated; Function `plot_confusion_matrix` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAW0UlEQVR4nO3de5hVdb3H8fdnBnDECygIhxCElDSkNA+p5ckHtafQetI6ZpkVmUVXLS2P2s2T5xyzp8zsYkVoahneSy3TOqQHLVPBW4KZBCgohAheQBRn5nv+WGtwgzCz1p69Z++15vN6nvWw19pr1voOPH78XdZFEYGZWRm1NLoAM7N6ccCZWWk54MystBxwZlZaDjgzK60BjS6g0qCB20XboKGNLsPyeH59oyuwHF5gHRviRfXmGG8/ZLt4anVHpn3nPfDizRExtTfn642mCri2QUM5cNInGl2G5RB3/7XRJVgOd8bsXh/jqdUd3HXz2Ez7to56ZHivT9gLTRVwZtb8Auiks9FlZOKAM7NcguClyNZFbTQHnJnl5hacmZVSEHQU5BZPB5yZ5daJA87MSiiADgecmZWVW3BmVkoBvOQxODMroyDcRTWzkgroKEa+OeDMLJ/kToZicMCZWU6ig17dr99nHHBmlksyyeCAM7MSSq6Dc8CZWUl1ugVnZmXkFpyZlVYgOgrytgMHnJnl5i6qmZVSIDZEa6PLyKQY7UwzaxrJhb4tmZaeSLpI0kpJD1Zs21nSHyQ9kv65U7pdkr4naaGkByTt19PxHXBmlltHerFvT0sGFwObv3XrdGB2REwAZqfrAIcDE9JlOvCjng7ugDOzXCJER7RkWno+VswBVm+2+UjgkvTzJcBRFdsvjcRfgKGSRnV3fI/BmVlundkvExkuaW7F+oyImNHDz4yMiOXp5xXAyPTzaGBpxX7L0m3L2QoHnJnlkkwyZI6OVRExuepzRYSkqp9d4oAzs1y6Jhnq6J+SRkXE8rQLujLd/jgwpmK/XdNtW+UxODPLrSOUaanS9cC09PM04LqK7R9OZ1MPBJ6p6MpukVtwZpZLLe9kkDQLmEIyVrcMOBM4B7hS0gnAo8Ax6e43AkcAC4HngeN7Or4Dzsxy68wwQ5pFRBy7la8O28K+AXwmz/EdcGaWS3KzfTFGtxxwZpZLIF4qyK1aDjgzyyWCTBfxNgMHnJnlpDwX+jaUA87McgncgjOzEvMkg5mVUiA/8NLMyil5bWAxoqMYVZpZE/GLn82spILa3clQbw44M8vNLTgzK6UIuQVnZuWUTDL4Vi0zKyX5Ql8zK6dkksFjcGZWUr6TwcxKyXcymFmp1fmlMzXjgDOzXCLgpU4HnJmVUNJFdcCZWUn5ToZ+aPjwdZz6+T8zdOgLEHDjzRO47jd7ccapt7Hrq54FYPvtNrB23SA+c/I7GlytAZzyncc44K3P8fSqAXzi0D0BeMs7n+ZDX1jBmAkvctIRE3jkgcENrrK5+DKRlKSpwPlAKzAzIs6p5/karbND/PSi/Vi4aBjbbvsS3z/3Ru69/1/4xrfesnGfjx8/j3XPD2xglVbp91fszPU/G86p5y/duG3J39o462PjOOmbyxpYWTMrThe1blVKagV+CBwOTASOlTSxXudrBqvXDGbhomEArF8/kKXLhjBs5/UVewQH/9uj3DpnXEPqs1d68M7teW7Npv+fX7qwjWX/aGtQRcXQmb6Xoael0erZgtsfWBgRiwAkXQ4cCSyo4zmbxsgRa9n91at5+O/DNm6bNHEla55u44nlOzawMrPeSWZRfS/qaGBpxfoy4IDNd5I0HZgO0DZoSB3L6TttbS/xldPm8JOZk3l+/aCN26ccvMStNyu8Il3o2/COdETMiIjJETF54IDtGl1Or7W2dvLV0+dwy/+N409/Gbtxe0tLJwe9aSlzbt+tgdWZ1Ya7qPA4MKZifdd0W4kFJ594B48tHcK112863PiGfVawdNmOrHqq+CFu/ZtnURN3AxMkjScJtvcDH6jj+Rpu79c+yVsPWcziJUP54Xm/BeDiX+zL3fNGM+UtS7j1tnGNLdBe4fQLHuX1b1rLkJ3b+cXcBfz83JE8t2YAn/7vxxkyrJ3/+vli/jG/jS9/YPdGl9pUijKLWreAi4h2SZ8Fbia5TOSiiJhfr/M1g/kPjWDqkR/c4nfnfu/NfVyNZXHOp7c8ZPDnm8oxHlwPEaK9vwccQETcCNxYz3OYWd9zF9XMSsljcGZWag44MyulIl0H54Azs9ya4Rq3LIoxFWJmTSMC2jtbMi09kXSypPmSHpQ0S1KbpPGS7pS0UNIVkgb1eKCtcMCZWW6doUxLdySNBk4CJkfEJJLLyd4PfBM4LyL2ANYAJ1RbpwPOzHLpGoPrbcClBgDbShoADAaWA4cCV6ffXwIcVW2tDjgzyy1CmRZguKS5Fcv0l48RjwPfBh4jCbZngHnA0xHRnu62jOTBHVXxJIOZ5ZZjkmFVREze0heSdiJ5hNp44GngKmBqLerr4oAzs1wianYd3FuBxRHxJICka4GDgKGSBqStuF49pMNdVDPLSXR0tmRaevAYcKCkwZIEHEbyQNxbgKPTfaYB11VbqQPOzHLLMQbXzTHiTpLJhHuAv5Lk0QzgNOAUSQuBYcCF1dbpLqqZ5VLLe1Ej4kzgzM02LyJ55UGvOeDMLJ9IxuGKwAFnZrkV5VYtB5yZ5RLpJEMROODMLDd3Uc2stHqaIW0WDjgzyyXCAWdmJeYHXppZaXkMzsxKKRCdnkU1s7IqSAPOAWdmOXmSwcxKrSBNOAecmeVW+BacpO/TTU5HxEl1qcjMmloAnZ0FDzhgbp9VYWbFEUDRW3ARcUnluqTBEfF8/Usys2ZXlOvgeryYRdKbJC0A/pau7yPpgrpXZmbNKzIuDZblar3vAm8HngKIiPuBg+tYk5k1tWyPK2+GiYhMs6gRsTR5J8RGHfUpx8wKoQlaZ1lkCbilkt4MhKSBwOeAh+pblpk1rYAoyCxqli7qJ4HPkLxd+glg33TdzPotZVwaq8cWXESsAo7rg1rMrCgK0kXNMov6akk3SHpS0kpJ10l6dV8UZ2ZNqkSzqL8ErgRGAa8CrgJm1bMoM2tiXRf6ZlkaLEvADY6In0dEe7r8Amird2Fm1rwisi2N1t29qDunH38n6XTgcpLsfh9wYx/UZmbNqiCzqN1NMswjCbSu3+QTFd8FcEa9ijKz5qYmaJ1l0d29qOP7shAzK4gmmUDIItOdDJImAROpGHuLiEvrVZSZNbPmmEDIoseAk3QmMIUk4G4EDgduBxxwZv1VQVpwWWZRjwYOA1ZExPHAPsCQulZlZs2tM+PSYFm6qOsjolNSu6QdgZXAmDrXZWbNqgwPvKwwV9JQ4KckM6trgTvqWZSZNbfCz6J2iYhPpx9/LOkmYMeIeKC+ZZlZUyt6wEnar7vvIuKe+pRkZv1F2jucCUwiic2PAg8DVwDjgCXAMRGxpprjd9eCO7eb7wI4tJoTdicGihd28V1gRXLrE/c1ugTLYf+31+a1KjXsop4P3BQRR0saBAwGvgTMjohz0ruoTgdOq+bg3V3oe0g1BzSzkgtqcquWpCEkrz/4CEBEbAA2SDqS5NI0gEuAW6ky4LJcJmJmtqnsj0saLmluxTK94ijjgSeBn0m6V9JMSdsBIyNiebrPCmBktWX6zfZmlluOLuqqiJi8le8GAPsBJ0bEnZLOJ+mObhQRIVXfIXYLzszyq80DL5cByyLiznT9apLA+6ekUQDpnyurLTPLE30l6YOSvpauj5W0f7UnNLMSqEHARcQKkpda7ZluOgxYAFwPTEu3TQOuq7bMLF3UC0huujgUOAt4DrgGeGO1JzWz4lLUdBb1ROCydAZ1EXA8ScPrSkknAI8Cx1R78CwBd0BE7CfpXoCIWJMWY2b9VY0eeBkR9wFbGqM7rBbHzxJwL0lqJW1wStqFpriN1swapSi3amWZZPge8CtghKT/IXlU0tl1rcrMmltB3qqV5V7UyyTNI2kyCjgqIvxme7P+qrZjcHWV5YGXY4HngRsqt0XEY/UszMyaWFkCDvgtL798po3k6uOHgb3rWJeZNTEVZBQ+Sxf1dZXr6VNGPr2V3c3MmkbuW7Ui4h5JB9SjGDMriLJ0USWdUrHaQnIrxRN1q8jMmluZJhmAHSo+t5OMyV1Tn3LMrBDKEHDpBb47RMQX+6geMyuCogecpAER0S7poL4syMyamyjHLOpdJONt90m6HrgKWNf1ZURcW+fazKwZlWwMrg14iuRpIl3XwwXggDPrr0oQcCPSGdQHeTnYuhTk1zOzuihIAnQXcK3A9mwabF0K8uuZWT2UoYu6PCLO6rNKzKw4ShBwtXminZmVS5RjFrUmT9Q0sxIqegsuIlb3ZSFmVhxlGIMzM9syB5yZlVKTPI48CwecmeUi3EU1sxJzwJlZeTngzKy0HHBmVkole5qImdmmHHBmVlZluFXLzGyL3EU1s3Lyhb5mVmoOODMrI9/JYGalps5iJJwDzszy8RicmZVZUbqoLY0uwMwKKDIuGUhqlXSvpN+k6+Ml3SlpoaQrJA2qtkwHnJnlpsi2ZPQ54KGK9W8C50XEHsAa4IRq63TAmVl+NWrBSdoVeAcwM10XyUvmr053uQQ4qtoyPQZnZvnke6vWcElzK9ZnRMSMivXvAv8B7JCuDwOejoj2dH0ZMLraUh1wZpZLzuvgVkXE5C0eR3onsDIi5kmaUpPiNuOAM7P8oibTqAcB75J0BNAG7AicDwyVNCBtxe0KPF7tCTwGZ2a51WKSISLOiIhdI2Ic8H7gjxFxHHALcHS62zTgumrrdMDV0KAB7fzoy9cx88xr+dnXr+Yj75q3yfcnHvtnfveDixtTnG107sljOOZ1ezP9kD03bptzwxA+PmVPpo7eh7/fv+3G7c+ubuXUo3fnyD1exw++VPVQULlknWCovpF3GnCKpIUkY3IXVnugunVRJV0EdPWxJ9XrPM1kQ3srp3z7CNa/OJDW1k6+f9oN3PXgGBYsGsGeuz3JDoM3NLpEA972vtW86/hVfOtzYzduG7fXC3xt5hK+d9qYTfYd1BZMO3UFSx5uY8nf2vq61KZV6+fBRcStwK3p50XA/rU4bj1bcBcDU+t4/CYk1r84EIABrZ0MaO0kAlrUySffexc/vrom/2bWS687cB077NSxybaxE15kzB4vvmLftsGdTDpgHYO2Kcil+31EndmWRqtbCy4i5kgaV6/jN6sWdTLjq79m9Ihn+dUtE3lo8Qj+/bAH+dP9Y1n9zOBGl2fWe0GtJhnqruFjcJKmS5orae5LG9Y1upxe64wWPnbWe3jvqcfy2vFP8voJy5kyeTG/mr13o0szq5ka38lQNw0PuIiYERGTI2LywEHbNbqcmlm7fhvu/dso3rDXckaPeJbLzr6Sy8+5nG0GtXPZ2Vc2ujyz3qnvJEPN+Dq4Ghqy/Xo6OlpYu34bBg1sZ/LEx5l10z685wvHbdzndz+4mOO+dEwDqzTrHT/wsp8aNvR5zvjoHFpaOmkR3HL3eO54YGzPP2h96huf2o0H7tieZ1YP4Lh/nciHvrCCHXbq4IKvjOaZpwbw1Q+9mt33Xs/ZsxYB8OH9J7JubQvtG8QdNw/h7Fn/YLfXvHJCot+I8AMvJc0CppDci7YMODMiqr6epQgWLRvGx896d7f7HP7Zj/RNMbZVZ/zo0S1uP+jwZ7a4/dK7FtSznGIqRr7VdRb12Hod28way11UMyunAPp7F9XMSqwY+eaAM7P83EU1s9Lq97OoZlZSTXIRbxYOODPLJbnQtxgJ54Azs/ya4EkhWTjgzCw3t+DMrJw8Bmdm5eV7Uc2szNxFNbNSyvfi54ZywJlZfm7BmVlpFSPfHHBmlp86i9FHdcCZWT6BL/Q1s3IS4Qt9zazEHHBmVloOODMrJY/BmVmZeRbVzEoq3EU1s5IKHHBmVmLF6KE64MwsP18HZ2blVZCAa2l0AWZWMBHQ0Zlt6YakMZJukbRA0nxJn0u37yzpD5IeSf/cqdpSHXBmll9EtqV77cAXImIicCDwGUkTgdOB2RExAZidrlfFAWdm+dUg4CJieUTck35+DngIGA0cCVyS7nYJcFS1ZXoMzszyCSD7OxmGS5pbsT4jImZsvpOkccAbgDuBkRGxPP1qBTCy2lIdcGaWU0Bkvk5kVURM7m4HSdsD1wCfj4hnJb18poiQVPWMhgPOzPIJepxAyErSQJJwuywirk03/1PSqIhYLmkUsLLa43sMzszyq8EYnJKm2oXAQxHxnYqvrgempZ+nAddVW6ZbcGaWX22ugzsI+BDwV0n3pdu+BJwDXCnpBOBR4JhqT+CAM7OcanOzfUTcDmgrXx/W6xPggDOzvALw45LMrLQKcquWA87McoqazaLWmwPOzPIJiOzXwTWUA87M8st+J0NDOeDMLD+PwZlZKUV4FtXMSswtODMrpyA6OhpdRCYOODPLJ9/jkhrKAWdm+fkyETMrowDCLTgzK6XI9cDLhnLAmVluRZlkUDTRdK+kJ0me/1Q2w4FVjS7Ccinrv9luEbFLbw4g6SaSv58sVkXE1N6crzeaKuDKStLcnp5Lb83F/2bl4EeWm1lpOeDMrLQccH3jFe+BtKbnf7MS8BicmZWWW3BmVloOODMrLQdcHUmaKulhSQslnd7oeqxnki6StFLSg42uxXrPAVcnklqBHwKHAxOBYyVNbGxVlsHFQMMuTLXacsDVz/7AwohYFBEbgMuBIxtck/UgIuYAqxtdh9WGA65+RgNLK9aXpdvMrI844MystBxw9fM4MKZifdd0m5n1EQdc/dwNTJA0XtIg4P3A9Q2uyaxfccDVSUS0A58FbgYeAq6MiPmNrcp6ImkWcAewp6Rlkk5odE1WPd+qZWal5RacmZWWA87MSssBZ2al5YAzs9JywJlZaTngCkRSh6T7JD0o6SpJg3txrIslHZ1+ntndgwAkTZH05irOsUTSK96+tLXtm+2zNue5/lPSF/PWaOXmgCuW9RGxb0RMAjYAn6z8UlJV77mNiI9FxIJudpkC5A44s0ZzwBXXbcAeaevqNknXAwsktUr6lqS7JT0g6RMASvwgfT7d/wIjug4k6VZJk9PPUyXdI+l+SbMljSMJ0pPT1uNbJO0i6Zr0HHdLOij92WGSfi9pvqSZgHr6JST9WtK89Gemb/bdeen22ZJ2SbftLumm9Gduk7RXTf42rZT8ZvsCSltqhwM3pZv2AyZFxOI0JJ6JiDdK2gb4k6TfA28A9iR5Nt1IYAFw0WbH3QX4KXBweqydI2K1pB8DayPi2+l+vwTOi4jbJY0luVvjtcCZwO0RcZakdwBZ7gL4aHqObYG7JV0TEU8B2wFzI+JkSV9Lj/1ZkpfBfDIiHpF0AHABcGgVf43WDzjgimVbSfeln28DLiTpOt4VEYvT7W8DXt81vgYMASYABwOzIqIDeELSH7dw/AOBOV3HioitPRftrcBEaWMDbUdJ26fneE/6s7+VtCbD73SSpHenn8ektT4FdAJXpNt/AVybnuPNwFUV594mwzmsn3LAFcv6iNi3ckP6H/q6yk3AiRFx82b7HVHDOlqAAyPihS3UkpmkKSRh+aaIeF7SrUDbVnaP9LxPb/53YLY1HoMrn5uBT0kaCCDpNZK2A+YA70vH6EYBh2zhZ/8CHCxpfPqzO6fbnwN2qNjv98CJXSuS9k0/zgE+kG47HNiph1qHAGvScNuLpAXZpQXoaoV+gKTr+yywWNJ703NI0j49nMP6MQdc+cwkGV+7J31xyk9IWuq/Ah5Jv7uU5IkZm4iIJ4HpJN3B+3m5i3gD8O6uSQbgJGByOomxgJdnc79OEpDzSbqqj/VQ603AAEkPAeeQBGyXdcD+6e9wKHBWuv044IS0vvn4MfDWDT9NxMxKyy04MystB5yZlZYDzsxKywFnZqXlgDOz0nLAmVlpOeDMrLT+H2nCfSiQO+rLAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig=plot_confusion_matrix(current_model, current_X_test_tf, current_y_test)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Second level: Comment classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_scorer = SentenceScorer(tf_idf_vectorizer=current_tf_idf, score_classifier_model=current_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-7-e1c77b292300>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:12: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
      "<ipython-input-7-e1c77b292300>:13: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Accuracy: 0.8\n",
      "Precision: 0.94\n",
      "Recall: 0.8\n",
      "F1 Score: 0.86\n",
      "AUC: 0.8\n"
     ]
    }
   ],
   "source": [
    "features = ['TextScore', 'Sentiment', 'CommentTextLen']\n",
    "\n",
    "f1_scores = []\n",
    "accuracy_scores = []\n",
    "precision_scores = []\n",
    "recall_scores = []\n",
    "auc_scores = []\n",
    "\n",
    "for train_index, test_index in sss.split(X, y):\n",
    "    X_train_two, X_test_two = X_two.iloc[train_index], X_two.iloc[test_index]\n",
    "    y_train_two, y_test_two = y_two[train_index], y_two[test_index]\n",
    "    X_train_two['TextScore'] = sentence_scorer.get_scores(X_train_two['CommentTextProc'])\n",
    "    X_test_two['TextScore'] = sentence_scorer.get_scores(X_test_two['CommentTextProc'])\n",
    "\n",
    "    brfc = BalancedRandomForestClassifier().fit(X_train_two[features], y_train_two)\n",
    "\n",
    "    y_predict = brfc.predict(X_test_two[features])\n",
    "\n",
    "    accuracy_scores.append(accuracy_score(y_test_two, y_predict))\n",
    "    precision_scores.append(precision_score(y_test_two, y_predict))\n",
    "    recall_scores.append(recall_score(y_test_two, y_predict))\n",
    "    f1_scores.append(f1_score(y_test_two,y_predict))\n",
    "    auc_scores.append(roc_auc_score(y_test_two,y_predict))\n",
    "    \n",
    "\n",
    "print(f\"\\n\\nAccuracy: {round(mean(accuracy_scores), 2)}\")\n",
    "print(f\"Precision: {round(mean(precision_scores), 2)}\")\n",
    "print(f\"Recall: {round(mean(recall_scores), 2)}\")\n",
    "print(f\"F1 Score: {round(mean(f1_scores), 2)}\")\n",
    "print(f\"AUC: {round(mean(auc_scores), 2)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function plot_confusion_matrix is deprecated; Function `plot_confusion_matrix` is deprecated in 1.0 and will be removed in 1.2. Use one of the class methods: ConfusionMatrixDisplay.from_predictions or ConfusionMatrixDisplay.from_estimator.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATgAAAEGCAYAAADxD4m3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaFklEQVR4nO3de5QV1Zn+8e/TjVwERJDLIGK8EQ0xioTgLTLeb9GFGmM0mYxRs9CoUWMmMyYzPzUmZkxiNCsmmtHAiMaIOmrEREUlzkIdL4CiAgYligIiyEUuokB3v78/TjUelT5d1fThnFP9fNaq1VX71Kl6aRYve9feu7YiAjOzPKqrdABmZuXiBGdmueUEZ2a55QRnZrnlBGdmudWp0gEU26pz9+jarXelw7AMtGptpUOwDD7gPdbHOm3ONY46pHssW96Y6tzpL66bFBFHb879NkdVJbiu3XqzzxcvqHQYlkGXB6dWOgTL4JmYvNnXWLa8kWcn7Zjq3PqBr/bd7BtuhqpKcGZW/QJooqnSYaTiBGdmmQTBhkjXRK00Jzgzy8w1ODPLpSBorJEpnk5wZpZZE05wZpZDATQ6wZlZXrkGZ2a5FMAGP4MzszwKwk1UM8upgMbayG9OcGaWTWEmQ21wgjOzjEQjmzVff4txgjOzTAqdDE5wZpZDhXFwTnBmllNNrsGZWR65BmdmuRWIxhpZ7cAJzswycxPVzHIpEOujvtJhpOIEZ2aZFAb6uolqZjnlTgYzy6UI0RiuwZlZTjW5BmdmeVToZKiN1FEbUZpZ1XAng5nlWqPHwZlZHtXSTIbaiNLMqkpT1KXaSpHUVdKzkl6QNEvSj5LynSU9I2mupDskdU7KuyTHc5PPd2otTic4M8ukMNm+LtXWinXAoRGxNzAMOFrSfsDPgGsjYjdgBXBWcv5ZwIqk/NrkvJKc4Mwsk0BsiPpUW8nrFKxJDrdKtgAOBf4nKR8PnJDsj06OST4/TFLJh4FOcGaWSQQ0Rl2qrTWS6iXNAJYAjwB/B96NiIbklAXAoGR/EDC/EEM0ACuB7Upd350MZpaRsgz07StpWtHxjRFxY/NBRDQCwyRtC9wL7NFuYeIEZ2YZBWSZqrU0Ika0es2IdyU9BuwPbCupU1JL2wFYmJy2EBgMLJDUCegFLCt1XTdRzSyz9uhkkNQvqbkhqRtwBPAy8BhwcnLa6cB9yf7E5Jjk879GRMkVWl2DM7NMArXXCy8HAuMl1VOobN0ZEX+WNBuYIOknwPPA2OT8scCtkuYCy4FTW7uBE5yZZVJYNnDzU0dEvAjss4ny14CRmyj/APhKlns4wZlZRl742cxyKqDVWQrVwgnOzDJzDc7McilCrsGZWT4VOhm8qpaZ5ZLXZDCznCp0MvgZnJnlVK288NIJzswyaceZDGXnBGdmmXnRGTPLpQjY0OQEZ2Y5VGiiOsGZWU55JkMH9eXDZ3LcqDlA8Jcpe/A/j+7JmSdM48BhbxAhVqzuxlXjRrHs3e6VDtWAi695k30PX827Sztx9qG7A7DL0Pf5zlUL6Na9icULOvOz83Zk7ZraGNi6JdTSMJGy1jMlHS1pTrLM1yXlvFc12HnQco4bNYdzfjKab11+Evvv/SaD+q9kwkN7cdblX+ZbPzqJp14YzOnHP1/pUC3x8B19+Pev7/yRsouuns+4nw7knMN258kHt+Hkby+pUHTVSu2ybOCWULYIkpfY/RY4BhgKnCZpaLnuVw12HPgus1/rx7r1nWhsqmPGnIEcNHweaz/ovPGcrl0aKP0OUtuSZj7Tg9UrPtqQ2WGXdbz0dKGG/fyUnnzxSysrEVpVa0rWZWhtq7RyptiRwNyIeC0i1gMTKCz7lVuvL+zNXkPeZpvuH9ClcwP77TWf/n3eA+CsE6dy5y9u54j9/s64P32+wpFaKW+80pX9j14FwEHHraTf9hsqHFF1KfSi1qfaKq2cCW7jEl+J4uW/NpI0RtI0SdM2rH+vjOGU35uLenP7g3vzi4sf5OfffYi5b/ahqanwv9jYe7/AKd8/jUee3pUTD5td4UitlGsuHszxpy/lNw+9QrcejTSsr3xNpJo0D/RNs1VaxTsZkiXEbgTo2WuHmm+8PfDE7jzwROFh9bdOmso7Kz7amfDo07vxs4smcfN9rsVVq/lzu/LD03YFYNAu69j3sFUVjqj6VEPzM41y1uCal/hqVrz8V25t2/N9APr3WcOo4fOY/PSuDOr/4TOcA4e9wZuLelUqPEuh13aFJqkUfO3Cxfz51pJrC3c4zb2oHb0GNxUYImlnContVOBrZbxfVbji3EfZpsc6Ghrr+NVtB7Dm/S58/4zH2fEfVtLUBIuX9eCaW79Y6TAtccn1b7DX/mvo1aeBP0ybza2/HEC3rZs4/ptLAXjywV48PKFPhaOsPtXQQ5pG2RJcRDRIOh+YBNQD4yJiVrnuVy0u+Nnxnyi77PrDKxCJpXHVuZ/aZPmfxvbbwpHUjgjR0NETHEBEPAA8UM57mNmWVw3NzzQq3slgZrXFMxnMLNfao5NB0mBJj0maLWmWpAuT8sslLZQ0I9mOLfrOD5KZUXMkHdVanK7BmVkm7fjCywbgexHxnKSewHRJjySfXRsRVxefnMyEOhX4LLA98KikT0dEY0s3cA3OzDJrj6laEbEoIp5L9lcDL7OJyQBFRgMTImJdRLwOzKUwY6pFTnBmlkkENDTVpdrSkrQTsA/wTFJ0vqQXJY2T1DspSzU7qpgTnJllluEZXN/mqZjJNubj15LUA7gbuCgiVgE3ALsCw4BFwC/bGqefwZlZJhmfwS2NiBEtfShpKwrJ7baIuAcgIhYXfX4T8OfkMPPsKNfgzCyzCKXaSpEkYCzwckRcU1Q+sOi0E4GZyf5E4FRJXZIZUkOAZ0vdwzU4M8usnSbbHwh8A3hJ0oyk7IcU3h05jMKQu3nA2QARMUvSncBsCj2w55XqQQUnODPLKKJ9BvpGxBOwyUzZ4uyniLgSuDLtPZzgzCwj0ehlA80sr1p7vlYtnODMLJNamovqBGdm2QQ1s3CSE5yZZVYrryx3gjOzTMKdDGaWZ26imlluuRfVzHIpwgnOzHLMw0TMLLf8DM7McikQTe5FNbO8qpEKnBOcmWXkTgYzy7UaqcI5wZlZZjVfg5N0HSXydERcUJaIzKyqBdDUVOMJDpi2xaIws9oRQK3X4CJifPGxpK0jYm35QzKzalcr4+BaHcwiaX9Js4G/Jcd7S7q+7JGZWfWKlFuFpRmt9yvgKGAZQES8AIwqY0xmVtXSLRlYDR0RqXpRI2J+YQnDjUou1WVmOVcFtbM00iS4+ZIOACJZhfpC4OXyhmVmVSsgaqQXNU0T9RzgPGAQ8BYwLDk2sw5LKbfKajXBRcTSiPh6RAyIiH4R8U8RsWxLBGdmVaodOhkkDZb0mKTZkmZJujAp7yPpEUmvJj97J+WS9GtJcyW9KGl4a2Gm6UXdRdL9kt6RtETSfZJ2ae17ZpZj7dOL2gB8LyKGAvsB50kaClwCTI6IIcDk5BjgGGBIso0BbmjtBmmaqH8E7gQGAtsDdwG3p/iemeVR80DfNFupy0Qsiojnkv3VFJ7tDwJGA83jcMcDJyT7o4FbouBpYFtJA0vdI02C2zoibo2IhmT7A9A1xffMLKci0m1pSdoJ2Ad4BhgQEYuSj94GBiT7g4D5RV9bkJS1qNRc1D7J7oOSLgEmUMjdXwUeSB+6meVO+l7UvpKKp33eGBE3Fp8gqQdwN3BRRKwqHpIWESGpzYNSSg0TmU4hoTXf7eyizwL4QVtvama1LUPKWRoRI1q8TmHo2d3AbRFxT1K8WNLAiFiUNEGXJOULgcFFX98hKWtRqbmoO6eJ3sw6mHaahqVCVW0s8HJEXFP00UTgdOCq5Od9ReXnS5oA7AusLGrKblKqmQyS9gSGUvTsLSJuSfnnMLNcab0DIaUDgW8AL0makZT9kEJiu1PSWcAbwCnJZw8AxwJzgbXAGa3doNUEJ+ky4GAKCe4BCl21TwBOcGYdVTvU4CLiCVoeDXzYJs4PMk4ySNOLenJys7cj4gxgb6BXlpuYWc40pdwqLE0T9f2IaJLUIGkbCg/8Brf2JTPLqTy88LLINEnbAjdR6FldAzxVzqDMrLq1feDGltVqgouIc5Pd30l6CNgmIl4sb1hmVtVqPcGVmsgqaXjzFAszs2pVqgb3yxKfBXBoO8cCAfXrq+DJpKU26a0ZlQ7BMhh5VPssq1LzTdSIOGRLBmJmNSLIMlWrorzws5llV+s1ODOzltR8E9XMrEU1kuDSvNFXkv5J0qXJ8Y6SRpY/NDOrWjlaF/V6YH/gtOR4NfDbskVkZlVNkX6rtDRN1H0jYrik5wEiYoWkzmWOy8yqWY56UTdIqiepcErqR1VMozWzSqmG2lkaaZqovwbuBfpLupLCq5J+WtaozKy61cgzuDRzUW+TNJ3CK5MEnBARXtnerKOqkudraaR54eWOFN6eeX9xWUS8Wc7AzKyK5SXBAX/hw8VnugI7A3OAz5YxLjOrYqqRp/BpmqifKz5O3jJybgunm5lVjcwzGSLiOUn7liMYM6sReWmiSrq46LAOGA68VbaIzKy65amTAehZtN9A4Znc3eUJx8xqQh4SXDLAt2dE/MsWisfMakGtJzhJnSKiQdKBWzIgM6tuonZ6UUvNZHg2+TlD0kRJ35B0UvO2JYIzsyrUjpPtJY2TtETSzKKyyyUtlDQj2Y4t+uwHkuZKmiPpqNaun+YZXFdgGYU1GJrHwwVwT4rvmlketV8T9WbgN8AtHyu/NiKuLi6QNBQ4lcIY3O2BRyV9OiIaW7p4qQTXP+lBncmHia1ZjbTAzaws2ikDRMQUSTulPH00MCEi1gGvS5oLjKTEOs2lmqj1QI9k61m037yZWQeVoYnaV9K0om1MylucL+nFpAnbOykbBMwvOmdBUtaiUjW4RRFxRcpgzKwjSV+DWxoRIzJe/Qbgx8ldfkxhCdMzM14DKJ3gauONdma2ZUV5e1EjYnHzvqSbgD8nhwuBwUWn7pCUtahUE/WwtgZoZjlXxvfBSRpYdHgihX4AgInAqZK6SNoZGMKHoz02qdTCz8vbFp6Z5V17TdWSdDtwMIVndQuAy4CDJQ2jkCLnAWcDRMQsSXcCsynMqjqvVA8qeNlAM2uL9utFPW0TxWNLnH8lcGXa6zvBmVk2VfI68jSc4MwsE5Gvt4mYmX2EE5yZ5ZcTnJnllhOcmeVSzt7oa2b2UU5wZpZXtfLCSyc4M8vMTVQzyycP9DWzXHOCM7M88kwGM8s1NdVGhnOCM7Ns/AzOzPLMTVQzyy8nODPLK9fgzCy/nODMLJfKvKpWe3KCM7NMPA7OzPItaiPDOcGZWWauwXVQXz56Jsf+4ysE8Pr83vz8poP4xb9NolvXDQBsu837zHmtH5f+6vDKBtqBrf9AfO+k3diwvo7GBjjoSyv55++/vfHz6/9jEJMm9OG+uS8BsHjBVlxz8Y6sXNaJnts28q/XvUG/7TdUKvzK80BfkDQOOA5YEhF7lus+1aRv7/c48cjZnPlvJ7F+Qyf+3/l/5dD9Xuein3xp4zmXXTCZ/5u+YwWjtK26BD+/6+90695Ewwa4+IQhfOHQVXzm82t55YVurFlZ/5Hzb7piEIefvJwjTlnBjCd68N//OZB/ve7NCkVfHWqlk6GujNe+GTi6jNevSvV1QZfOjdTVNdG1cyNLV2y98bOtu65nn6GLeHL6pyoYoUnQrXvhX2jDBtG4QUjQ2Ag3/Xh7zvqPtz5y/huvdGHvA9cAsPeBa3hqUq8tHnO1UVO6rdXrSOMkLZE0s6isj6RHJL2a/OydlEvSryXNlfSipOGtXb9sCS4ipgDLy3X9arR0RXfuemBPbv/VHdx13QTWvL8V02cO2vj5gZ9/g+dnbc/aDzpXMEqDQjL79uG789W99mSfUavZY/haJv53X/Y/chXbDWj4yLm7DP2AJx8sJLUnH+zF2jX1rFpev6nLdgxBoZMhzda6m/lkRegSYHJEDAEmJ8cAxwBDkm0McENrFy9nDS4VSWMkTZM0bcOG9yodzmbpsfU6Dvj8m3z94q9wygWn0q1LA4cfMHfj54fu/xp/fWqXCkZozerr4YZH53Db9NnMmbE1Lz3dncfv35bRZ77ziXPHXLqQl57qwblHfJqXnupB34HrqevA+Q0KnQxptta0UBEaDYxP9scDJxSV3xIFTwPbShpY6voVT3ARcWNEjIiIEVtt1b3S4WyW4Xu+xdvv9GDl6m40Ntbx+NRPMXTIEgC26fEBe+yylKdf2KHCUVqxHr0a2fuANbzwZA/emteFMw4Yyj+PHMq69+v45gGfAWC7f2jg0rHzuP6RV/jmJYs2fq9Di5Qb9G2uwCTbmBRXHxARi5L9t4EByf4gYH7ReQuSsha5F7UdLVnWnc/s+g5dOjewbn09wz+7iDmv9wVg1Mh5PD1jMBs2+Fdeae8uq6dTp0KSWve+eG5KT045bwkTXpi18ZzRu32Om//vZQBWLqunZ+9G6upgwnX9OfKrHerJyydkHOi7NCJGtPVeERFS2wel+F9bO/rb3/szZepO/O7H99HYJObO246/PLY7AIfs9xoT7t+rwhEawPLFW3H1hTvS1CSammDU8e+y3xGrWjz/xad6MO4/t0cKPrfve5z30wVbMNoqFFHuF14uljQwIhYlTdAlSflCYHDReTskZS0q5zCR24GDKVRRFwCXRcTYct2vWoy/Zzjj7/lk5873fnpsBaKxTdll6Adc/8grJc9pHgMHcNBxKznouJXlDqu2lHcc3ETgdOCq5Od9ReXnS5oA7AusLGrKblLZElxEnFaua5tZZbXXTIZNVYQoJLY7JZ0FvAGckpz+AHAsMBdYC5zR2vXdRDWzbAJopyZqiYrQYZs4N4DzslzfCc7MsuvoU7XMLL882d7McsvLBppZPvltImaWV4WBvrWR4ZzgzCy7GnldkhOcmWXmGpyZ5ZOfwZlZfpV9Lmq7cYIzs+zcRDWzXPLCz2aWa67BmVlu1UZ+c4Izs+zUVBttVCc4M8sm8EBfM8snER7oa2Y55gRnZrnlBGdmueRncGaWZ+5FNbOcCjdRzSynAic4M8ux2mihOsGZWXYeB2dm+dVOCU7SPGA10Ag0RMQISX2AO4CdgHnAKRGxoi3Xr2uXKM2s44iAxqZ0WzqHRMSwiBiRHF8CTI6IIcDk5LhNnODMLLuIdFvbjAbGJ/vjgRPaeiEnODPLLn2C6ytpWtE25uNXAh6WNL3oswERsSjZfxsY0NYw/QzOzLIJIP2aDEuLmp6b8sWIWCipP/CIpL995FYRIanNVUHX4Mwso4BoSre1dqWIhcnPJcC9wEhgsaSBAMnPJW2N1AnOzLIJ2qWTQVJ3ST2b94EjgZnAROD05LTTgfvaGqqbqGaWXfsMExkA3CsJCrnojxHxkKSpwJ2SzgLeAE5p6w2c4Mwsu3ZIcBHxGrD3JsqXAYdt9g1wgjOzzDzZ3szyKgC/LsnMcss1ODPLp8gyDauinODMLJuASDHGrRo4wZlZdulnMlSUE5yZZedncGaWSxHuRTWzHHMNzszyKYjGxkoHkYoTnJllk+11SRXlBGdm2XmYiJnlUQDhGpyZ5VKEa3Bmll+10smgqKLuXknvUHjBXd70BZZWOgjLJK9/Z5+KiH6bcwFJD1H4/aSxNCKO3pz7bY6qSnB5JWlaKwtvWJXx31k+eE0GM8stJzgzyy0nuC3jxkoHYJn57ywH/AzOzHLLNTgzyy0nODPLLSe4MpJ0tKQ5kuZKuqTS8VjrJI2TtETSzErHYpvPCa5MJNUDvwWOAYYCp0kaWtmoLIWbgYoNTLX25QRXPiOBuRHxWkSsByYAoysck7UiIqYAyysdh7UPJ7jyGQTMLzpekJSZ2RbiBGdmueUEVz4LgcFFxzskZWa2hTjBlc9UYIiknSV1Bk4FJlY4JrMOxQmuTCKiATgfmAS8DNwZEbMqG5W1RtLtwFPA7pIWSDqr0jFZ23mqlpnllmtwZpZbTnBmlltOcGaWW05wZpZbTnBmlltOcDVEUqOkGZJmSrpL0tabca2bJZ2c7P++1IsAJB0s6YA23GOepE+svtRS+cfOWZPxXpdL+pesMVq+OcHVlvcjYlhE7AmsB84p/lBSm9a5jYhvRcTsEqccDGROcGaV5gRXux4HdktqV49LmgjMllQv6ReSpkp6UdLZACr4TfJ+ukeB/s0XkvS/kkYk+0dLek7SC5ImS9qJQiL9blJ7PEhSP0l3J/eYKunA5LvbSXpY0ixJvwfU2h9C0p8kTU++M+Zjn12blE+W1C8p21XSQ8l3Hpe0R7v8Ni2XvLJ9DUpqascADyVFw4E9I+L1JEmsjIgvSOoCPCnpYWAfYHcK76YbAMwGxn3suv2Am4BRybX6RMRySb8D1kTE1cl5fwSujYgnJO1IYbbGZ4DLgCci4gpJXwLSzAI4M7lHN2CqpLsjYhnQHZgWEd+VdGly7fMpLAZzTkS8Kmlf4Hrg0Db8Gq0DcIKrLd0kzUj2HwfGUmg6PhsRryflRwJ7NT9fA3oBQ4BRwO0R0Qi8Jemvm7j+fsCU5mtFREvvRTscGCptrKBtI6lHco+Tku/+RdKKFH+mCySdmOwPTmJdBjQBdyTlfwDuSe5xAHBX0b27pLiHdVBOcLXl/YgYVlyQ/EN/r7gI+E5ETPrYece2Yxx1wH4R8cEmYklN0sEUkuX+EbFW0v8CXVs4PZL7vvvx34FZS/wMLn8mAd+WtBWApE9L6g5MAb6aPKMbCByyie8+DYyStHPy3T5J+WqgZ9F5DwPfaT6QNCzZnQJ8LSk7BujdSqy9gBVJctuDQg2yWR3QXAv9GoWm7yrgdUlfSe4hSXu3cg/rwJzg8uf3FJ6vPZcsnPJfFGrq9wKvJp/dQuGNGR8REe8AYyg0B1/gwybi/cCJzZ0MwAXAiKQTYzYf9ub+iEKCnEWhqfpmK7E+BHSS9DJwFYUE2+w9YGTyZzgUuCIp/zpwVhLfLPwaeCvBbxMxs9xyDc7McssJzsxyywnOzHLLCc7McssJzsxyywnOzHLLCc7Mcuv/A2qkMuBw6GgKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig=plot_confusion_matrix(brfc, X_test_two[features], y_test_two)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save scorer and classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['comment_classifier.pkl']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "joblib.dump(sentence_scorer, 'sentence_scorer.pkl', compress=9)\n",
    "joblib.dump(brfc, 'comment_classifier.pkl', compress=9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}