parts_of_speech_analysis.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Notebook for Parts of Speech Analysis\n",
        "\n",
        "Using spaCy for parts of speech analysis, we want to create relative frequency tables for the parts of speech by year.\n",
        "\n",
        "Currently processes the \"Fakespeak-ENG modified.xlsx\" file (I've renamed my copy to \"Fakespeak_ENG_modified.xlsx\" to create a more consistent path), but will eventually be run on data from MisInfoText as well.\n",
        "\n",
        "From the original data file, we use the following columns: ID, combinedLabel, originalTextType, originalBodyText, originalDateYear\n",
        "\n",
        "We are processing text from the \"originalBodyText\" column."
      ],
      "metadata": {
        "id": "gPFsr_9hlORo"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2nH7fSVElJ8g"
      },
      "outputs": [],
      "source": [
        "import spacy\n",
        "import pandas as pd"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install xlsxwriter # for writing to multiple excel sheets"
      ],
      "metadata": {
        "id": "LxSjBsHooEDv"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "id": "CRlhU7N8ny77"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Loading articles into dataframes, separated by year"
      ],
      "metadata": {
        "id": "_uvTFrdcslth"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "input = '/content/drive/My Drive/fake_news_over_time/Fakespeak_ENG_modified.xlsx'"
      ],
      "metadata": {
        "id": "0lBbCE97n7C6"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fakespeak_df = pd.read_excel(input, sheet_name=\"Working\", usecols=['ID', 'combinedLabel', 'originalTextType', 'originalBodyText', 'originalDateYear'])"
      ],
      "metadata": {
        "id": "1j_Eh7yNs19t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "fakespeak_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "1teGoy-_xcMJ",
        "outputId": "4544d771-dd5b-4daa-dced-4b81f6f23e53"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                     ID combinedLabel originalTextType  \\\n",
              "0  Politifact_FALSE_Social media_687276         False     Social media   \n",
              "1   Politifact_FALSE_Social media_25111         False     Social media   \n",
              "2  Politifact_FALSE_Social media_735424         False     Social media   \n",
              "3  Politifact_FALSE_Social media_594307         False     Social media   \n",
              "4  Politifact_FALSE_Social media_839325         False     Social media   \n",
              "\n",
              "                                    originalBodyText  originalDateYear  \n",
              "0  Mexico is paying for the Wall through the new ...              2019  \n",
              "1  Chuck Schumer: \"why should American citizens b...              2019  \n",
              "2  Billions of dollars are sent to the State of C...              2019  \n",
              "3  If 50 Billion $$ were set aside to go towards ...              2019  \n",
              "4  Huge@#CD 9 news. \\n@ncsbe\\n sent letter to eve...              2019  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-18dcc346-8aef-4ef7-afd3-f28f1812e99b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ID</th>\n",
              "      <th>combinedLabel</th>\n",
              "      <th>originalTextType</th>\n",
              "      <th>originalBodyText</th>\n",
              "      <th>originalDateYear</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Politifact_FALSE_Social media_687276</td>\n",
              "      <td>False</td>\n",
              "      <td>Social media</td>\n",
              "      <td>Mexico is paying for the Wall through the new ...</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Politifact_FALSE_Social media_25111</td>\n",
              "      <td>False</td>\n",
              "      <td>Social media</td>\n",
              "      <td>Chuck Schumer: \"why should American citizens b...</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Politifact_FALSE_Social media_735424</td>\n",
              "      <td>False</td>\n",
              "      <td>Social media</td>\n",
              "      <td>Billions of dollars are sent to the State of C...</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Politifact_FALSE_Social media_594307</td>\n",
              "      <td>False</td>\n",
              "      <td>Social media</td>\n",
              "      <td>If 50 Billion $$ were set aside to go towards ...</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Politifact_FALSE_Social media_839325</td>\n",
              "      <td>False</td>\n",
              "      <td>Social media</td>\n",
              "      <td>Huge@#CD 9 news. \\n@ncsbe\\n sent letter to eve...</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-18dcc346-8aef-4ef7-afd3-f28f1812e99b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-18dcc346-8aef-4ef7-afd3-f28f1812e99b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-18dcc346-8aef-4ef7-afd3-f28f1812e99b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c170cf2e-c8bf-4809-8455-0cb4710c6bae\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c170cf2e-c8bf-4809-8455-0cb4710c6bae')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c170cf2e-c8bf-4809-8455-0cb4710c6bae button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "fakespeak_df",
              "summary": "{\n  \"name\": \"fakespeak_df\",\n  \"rows\": 2961,\n  \"fields\": [\n    {\n      \"column\": \"ID\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2960,\n        \"samples\": [\n          \"Politifact_Mostly False_Social media_959910\",\n          \"Politifact_FALSE_Social media_761958\",\n          \"Politifact_FALSE_Social media_565930\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"combinedLabel\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"False\",\n          \"Mostly False\",\n          \"Pants on Fire\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"originalTextType\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"Social media\",\n          \"News and blog\",\n          \"Press release\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"originalBodyText\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 2958,\n        \"samples\": [\n          \"Biden's Capital Gains Tax means that when you sell \\u00abbur home you'll owe taxes of\\n40% of your profit!\\nLet that sink info!\\n5#8%!\",\n          \"In this video we will see farmers getting letters from the government to destroy their crops.\",\n          \"Talk about indoctrinating kids. Turning Point has now recruited junior high kids at Pleasant Valley.  \\n@Iowans4PublicEd\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"originalDateYear\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1,\n        \"min\": 2019,\n        \"max\": 2024,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          2019,\n          2020,\n          2024\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Tagging parts of speech using spaCy"
      ],
      "metadata": {
        "id": "FVjz0URmxhZB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Using the small English web model, we tag the parts of speech in the body text by making article's body text a string, analyzing the string using spaCy, and then appending each token to a list manually.\n",
        "\n",
        "We end up with a dataframe of many rows since each tag/tagged token takes up one row - this is fine since we are looking at overall counts in a year and we don't need to preserve the delineation between articles."
      ],
      "metadata": {
        "id": "iHjWNe6IAEDx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create the spacy model\n",
        "nlp = spacy.load(\"en_core_web_sm\")\n",
        "\n",
        "# lists to hold spacy output elements\n",
        "tokens = []\n",
        "pos = []\n",
        "years = []\n",
        "\n",
        "for index, row in fakespeak_df.iterrows():\n",
        "  doc = nlp(row['originalBodyText'])\n",
        "  year = row['originalDateYear']\n",
        "\n",
        "  for token in doc:\n",
        "    tokens.append(token.text)\n",
        "    pos.append(token.pos_)\n",
        "    years.append(year)"
      ],
      "metadata": {
        "id": "nb3z_hLE8Kxe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "At this point, we are testing the code with just articles from 2019, so pos_df holds the tokens and tags for 2019 articles specifically as shown below."
      ],
      "metadata": {
        "id": "oWP1isOiArka"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# dictionary to map the lists\n",
        "tags = {\n",
        "    'Token': tokens,\n",
        "    'POS': pos,\n",
        "    'Year': years\n",
        "}\n",
        "\n",
        "# create a new dataframe containing the pos\n",
        "pos_df = pd.DataFrame(tags)"
      ],
      "metadata": {
        "id": "0xcpgghinEKR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pos_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "YgttJcJiC7gx",
        "outputId": "44101b71-dee3-4644-c233-a64178f50ee8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "    Token    POS  Year\n",
              "0  Mexico  PROPN  2019\n",
              "1      is    AUX  2019\n",
              "2  paying   VERB  2019\n",
              "3     for    ADP  2019\n",
              "4     the    DET  2019"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-5367c801-955b-4fa7-a7bc-8ba904ff1d9e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Token</th>\n",
              "      <th>POS</th>\n",
              "      <th>Year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Mexico</td>\n",
              "      <td>PROPN</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>is</td>\n",
              "      <td>AUX</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>paying</td>\n",
              "      <td>VERB</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>for</td>\n",
              "      <td>ADP</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>the</td>\n",
              "      <td>DET</td>\n",
              "      <td>2019</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-5367c801-955b-4fa7-a7bc-8ba904ff1d9e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-5367c801-955b-4fa7-a7bc-8ba904ff1d9e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-5367c801-955b-4fa7-a7bc-8ba904ff1d9e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-514dc4cd-372d-468b-a54b-f3c1ae409583\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-514dc4cd-372d-468b-a54b-f3c1ae409583')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-514dc4cd-372d-468b-a54b-f3c1ae409583 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "pos_df"
            }
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create relative frequency tables of parts of speech by year"
      ],
      "metadata": {
        "id": "YznSANUdOqfu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# do this for 2019 to start\n",
        "pos_2019_df = pos_df[pos_df['Year'] == 2019]\n",
        "\n",
        "counts_19 = pos_2019_df['POS'].value_counts()\n",
        "total_19 = counts_19.sum()\n",
        "\n",
        "print(counts_19)\n",
        "print(\"Total number of tokens: \", total_19)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dUZlCZFvOtf0",
        "outputId": "63306aed-09f1-4693-8cf4-517378a721cd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "POS\n",
            "NOUN     6641\n",
            "VERB     4168\n",
            "PUNCT    4099\n",
            "PROPN    3557\n",
            "ADP      3551\n",
            "DET      2708\n",
            "PRON     2502\n",
            "ADJ      2175\n",
            "AUX      1915\n",
            "ADV      1217\n",
            "CCONJ    1085\n",
            "PART     1055\n",
            "SPACE    1020\n",
            "NUM       744\n",
            "SCONJ     670\n",
            "SYM       133\n",
            "X          60\n",
            "INTJ       52\n",
            "Name: count, dtype: int64\n",
            "Total number of tokens:  37352\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "counts_19.plot(kind='bar')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 500
        },
        "id": "iJb0Eeo3PxDp",
        "outputId": "8d23037c-1030-4ad6-b93f-d5906459c14d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<Axes: xlabel='POS'>"
            ]
          },
          "metadata": {},
          "execution_count": 13
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAHSCAYAAADylfF7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABNAUlEQVR4nO3deVhU5f8//ucMMIDAgJKCJiqmIKSGW0iauSCoaC70MVxRcH2DKZZbmQuZuLxdMNcSAStNKTc0NcXccSNx35cgcdB3CqOkIHD//ujL/BwFFTgzeOD5uK5zXc4599yvczwsT87c5z4KIYQAERERkYwoy3oHiIiIiIqLAYaIiIhkhwGGiIiIZIcBhoiIiGSHAYaIiIhkhwGGiIiIZIcBhoiIiGTHtKx3wFDy8/ORlpYGGxsbKBSKst4dIiIiegVCCDx48AA1atSAUln0dZZyG2DS0tLg5ORU1rtBREREJZCamoqaNWsWub3cBhgbGxsA//4HqNXqMt4bIiIiehVarRZOTk663+NFKbcBpuBjI7VazQBDREQkMy8b/sFBvERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDumZb0DZaXOxG0let/NWX4S7wkREREVF6/AEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsFDvA3Lp1C/3794e9vT0sLS3RqFEjnDhxQrddCIEpU6agevXqsLS0hLe3N65cuaLXx71799CvXz+o1WrY2dkhODgYDx8+1Gtz+vRpvP/++7CwsICTkxPmzJlTwkMkIiKi8qZYAeb+/fto1aoVzMzMsH37dpw/fx7z5s1D5cqVdW3mzJmDRYsWYfny5Th69CisrKzg6+uLx48f69r069cP586dw65du7B161bs378fw4YN023XarXw8fFB7dq1kZSUhLlz52LatGn49ttvJThkIiIikjuFEEK8auOJEyfi0KFDOHDgQKHbhRCoUaMGPv30U3z22WcAgMzMTDg4OCAmJgYBAQG4cOEC3N3dcfz4cTRv3hwAsGPHDnTp0gV//fUXatSogWXLluGLL76ARqOBSqXS1d60aRMuXrz4Svuq1Wpha2uLzMxMqNXq57bXmbjtVQ9bz81ZfiV6HxEREb3cy35/FyjWFZgtW7agefPm+L//+z9Uq1YNTZo0wXfffafbfuPGDWg0Gnh7e+vW2drawtPTE4mJiQCAxMRE2NnZ6cILAHh7e0OpVOLo0aO6Nm3atNGFFwDw9fXFpUuXcP/+/UL3LTs7G1qtVm8hIiKi8qlYAeb69etYtmwZ6tevj507d2LkyJH45JNPEBsbCwDQaDQAAAcHB733OTg46LZpNBpUq1ZNb7upqSmqVKmi16awPp6u8ayIiAjY2trqFicnp+IcGhEREclIsQJMfn4+mjZtipkzZ6JJkyYYNmwYhg4diuXLlxtq/17ZpEmTkJmZqVtSU1PLepeIiIjIQIoVYKpXrw53d3e9dW5ubkhJSQEAODo6AgDS09P12qSnp+u2OTo64s6dO3rbc3Nzce/ePb02hfXxdI1nmZubQ61W6y1ERERUPhUrwLRq1QqXLl3SW3f58mXUrl0bAODs7AxHR0ckJCTotmu1Whw9ehReXl4AAC8vL2RkZCApKUnXZs+ePcjPz4enp6euzf79+/HkyRNdm127dsHV1VXvjiciIiKqmIoVYMLCwnDkyBHMnDkTV69exZo1a/Dtt98iJCQEAKBQKDBmzBjMmDEDW7ZswZkzZzBw4EDUqFEDPXr0APDvFZtOnTph6NChOHbsGA4dOoTQ0FAEBASgRo0aAIC+fftCpVIhODgY586dw7p16xAZGYmxY8dKe/REREQkS6bFadyiRQts3LgRkyZNQnh4OJydnbFw4UL069dP12b8+PHIysrCsGHDkJGRgdatW2PHjh2wsLDQtfnxxx8RGhqKDh06QKlUwt/fH4sWLdJtt7W1xW+//YaQkBA0a9YMb7zxBqZMmaI3VwwRERFVXMWaB0ZOOA8MERGR/BhkHhgiIiKi1wEDDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJTrECzLRp06BQKPSWBg0a6LY/fvwYISEhsLe3h7W1Nfz9/ZGenq7XR0pKCvz8/FCpUiVUq1YN48aNQ25url6bvXv3omnTpjA3N0e9evUQExNT8iMkIiKicqfYV2Defvtt3L59W7ccPHhQty0sLAzx8fGIi4vDvn37kJaWhl69eum25+Xlwc/PDzk5OTh8+DBiY2MRExODKVOm6NrcuHEDfn5+aNeuHZKTkzFmzBgMGTIEO3fuLOWhEhERUXlhWuw3mJrC0dHxufWZmZmIiorCmjVr0L59ewBAdHQ03NzccOTIEbRs2RK//fYbzp8/j927d8PBwQEeHh746quvMGHCBEybNg0qlQrLly+Hs7Mz5s2bBwBwc3PDwYMHsWDBAvj6+pbycImIiKg8KPYVmCtXrqBGjRqoW7cu+vXrh5SUFABAUlISnjx5Am9vb13bBg0aoFatWkhMTAQAJCYmolGjRnBwcNC18fX1hVarxblz53Rtnu6joE1BH0XJzs6GVqvVW4iIiKh8KlaA8fT0RExMDHbs2IFly5bhxo0beP/99/HgwQNoNBqoVCrY2dnpvcfBwQEajQYAoNFo9MJLwfaCbS9qo9Vq8ejRoyL3LSIiAra2trrFycmpOIdGREREMlKsj5A6d+6s+3fjxo3h6emJ2rVrY/369bC0tJR854pj0qRJGDt2rO61VqtliCEiIiqnSnUbtZ2dHVxcXHD16lU4OjoiJycHGRkZem3S09N1Y2YcHR2fuyup4PXL2qjV6heGJHNzc6jVar2FiIiIyqdSBZiHDx/i2rVrqF69Opo1awYzMzMkJCTotl+6dAkpKSnw8vICAHh5eeHMmTO4c+eOrs2uXbugVqvh7u6ua/N0HwVtCvogIiIiKlaA+eyzz7Bv3z7cvHkThw8fRs+ePWFiYoI+ffrA1tYWwcHBGDt2LH7//XckJSVh8ODB8PLyQsuWLQEAPj4+cHd3x4ABA3Dq1Cns3LkTkydPRkhICMzNzQEAI0aMwPXr1zF+/HhcvHgRS5cuxfr16xEWFib90RMREZEsFWsMzF9//YU+ffrg77//RtWqVdG6dWscOXIEVatWBQAsWLAASqUS/v7+yM7Ohq+vL5YuXap7v4mJCbZu3YqRI0fCy8sLVlZWCAwMRHh4uK6Ns7Mztm3bhrCwMERGRqJmzZpYuXIlb6EmIiIiHYUQQpT1ThiCVquFra0tMjMzCx0PU2fithL1e3OWX2l3jYiIiIrwst/fBfgsJCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpId07LegYqgzsRtJXrfzVl+Eu8JERFR+cArMERERCQ7DDBEREQkOwwwREREJDscA1MOccwNERGVd6W6AjNr1iwoFAqMGTNGt+7x48cICQmBvb09rK2t4e/vj/T0dL33paSkwM/PD5UqVUK1atUwbtw45Obm6rXZu3cvmjZtCnNzc9SrVw8xMTGl2VUiIiIqR0ocYI4fP44VK1agcePGeuvDwsIQHx+PuLg47Nu3D2lpaejVq5due15eHvz8/JCTk4PDhw8jNjYWMTExmDJliq7NjRs34Ofnh3bt2iE5ORljxozBkCFDsHPnzpLuLhEREZUjJQowDx8+RL9+/fDdd9+hcuXKuvWZmZmIiorC/Pnz0b59ezRr1gzR0dE4fPgwjhw5AgD47bffcP78efzwww/w8PBA586d8dVXX2HJkiXIyckBACxfvhzOzs6YN28e3NzcEBoaio8++ggLFiyQ4JCJiIhI7koUYEJCQuDn5wdvb2+99UlJSXjy5Ine+gYNGqBWrVpITEwEACQmJqJRo0ZwcHDQtfH19YVWq8W5c+d0bZ7t29fXV9dHYbKzs6HVavUWIiIiKp+KPYj3p59+wh9//IHjx48/t02j0UClUsHOzk5vvYODAzQaja7N0+GlYHvBthe10Wq1ePToESwtLZ+rHRERgenTpxf3cIiIiEiGinUFJjU1FaNHj8aPP/4ICwsLQ+1TiUyaNAmZmZm6JTU1tax3iYiIiAykWAEmKSkJd+7cQdOmTWFqagpTU1Ps27cPixYtgqmpKRwcHJCTk4OMjAy996Wnp8PR0REA4Ojo+NxdSQWvX9ZGrVYXevUFAMzNzaFWq/UWIiIiKp+KFWA6dOiAM2fOIDk5Wbc0b94c/fr10/3bzMwMCQkJuvdcunQJKSkp8PLyAgB4eXnhzJkzuHPnjq7Nrl27oFar4e7urmvzdB8FbQr6ICIiooqtWGNgbGxs0LBhQ711VlZWsLe3160PDg7G2LFjUaVKFajVaowaNQpeXl5o2bIlAMDHxwfu7u4YMGAA5syZA41Gg8mTJyMkJATm5uYAgBEjRmDx4sUYP348goKCsGfPHqxfvx7btpVsgjYiIiIqXySfiXfBggVQKpXw9/dHdnY2fH19sXTpUt12ExMTbN26FSNHjoSXlxesrKwQGBiI8PBwXRtnZ2ds27YNYWFhiIyMRM2aNbFy5Ur4+vpKvbtEREQkQ6UOMHv37tV7bWFhgSVLlmDJkiVFvqd27dr49ddfX9hv27ZtcfLkydLuHhEREZVDfJgjERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREcmOaVnvAMlbnYnbSvS+m7P8ZFGPiIheT7wCQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREssMAQ0RERLLDAENERESywwBDREREslOsALNs2TI0btwYarUaarUaXl5e2L59u27748ePERISAnt7e1hbW8Pf3x/p6el6faSkpMDPzw+VKlVCtWrVMG7cOOTm5uq12bt3L5o2bQpzc3PUq1cPMTExJT9CIiIiKneKFWBq1qyJWbNmISkpCSdOnED79u3RvXt3nDt3DgAQFhaG+Ph4xMXFYd++fUhLS0OvXr1078/Ly4Ofnx9ycnJw+PBhxMbGIiYmBlOmTNG1uXHjBvz8/NCuXTskJydjzJgxGDJkCHbu3CnRIRMREZHcmRancbdu3fRef/3111i2bBmOHDmCmjVrIioqCmvWrEH79u0BANHR0XBzc8ORI0fQsmVL/Pbbbzh//jx2794NBwcHeHh44KuvvsKECRMwbdo0qFQqLF++HM7Ozpg3bx4AwM3NDQcPHsSCBQvg6+sr0WETERGRnJV4DExeXh5++uknZGVlwcvLC0lJSXjy5Am8vb11bRo0aIBatWohMTERAJCYmIhGjRrBwcFB18bX1xdarVZ3FScxMVGvj4I2BX0UJTs7G1qtVm8hIiKi8qnYAebMmTOwtraGubk5RowYgY0bN8Ld3R0ajQYqlQp2dnZ67R0cHKDRaAAAGo1GL7wUbC/Y9qI2Wq0Wjx49KnK/IiIiYGtrq1ucnJyKe2hEREQkE8UOMK6urkhOTsbRo0cxcuRIBAYG4vz584bYt2KZNGkSMjMzdUtqampZ7xIREREZSLHGwACASqVCvXr1AADNmjXD8ePHERkZiY8//hg5OTnIyMjQuwqTnp4OR0dHAICjoyOOHTum11/BXUpPt3n2zqX09HSo1WpYWloWuV/m5uYwNzcv7uEQERGRDJV6Hpj8/HxkZ2ejWbNmMDMzQ0JCgm7bpUuXkJKSAi8vLwCAl5cXzpw5gzt37uja7Nq1C2q1Gu7u7ro2T/dR0KagDyIiIqJiXYGZNGkSOnfujFq1auHBgwdYs2YN9u7di507d8LW1hbBwcEYO3YsqlSpArVajVGjRsHLywstW7YEAPj4+MDd3R0DBgzAnDlzoNFoMHnyZISEhOiunowYMQKLFy/G+PHjERQUhD179mD9+vXYtm2b9EdPREREslSsAHPnzh0MHDgQt2/fhq2tLRo3boydO3eiY8eOAIAFCxZAqVTC398f2dnZ8PX1xdKlS3XvNzExwdatWzFy5Eh4eXnBysoKgYGBCA8P17VxdnbGtm3bEBYWhsjISNSsWRMrV67kLdRERESkU6wAExUV9cLtFhYWWLJkCZYsWVJkm9q1a+PXX399YT9t27bFyZMni7NrREREVIHwWUhEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7xX4aNVFFUmdi8Z/BdXOWnwH2hIiInsYrMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDuciZfoNVGSWX8BzvxLRBUTr8AQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMAQERGR7DDAEBERkeyYlvUOEFHZqDNxW7Hfc3OWnwH2hIio+HgFhoiIiGSHAYaIiIhkhwGGiIiIZIcBhoiIiGSnWAEmIiICLVq0gI2NDapVq4YePXrg0qVLem0eP36MkJAQ2Nvbw9raGv7+/khPT9drk5KSAj8/P1SqVAnVqlXDuHHjkJubq9dm7969aNq0KczNzVGvXj3ExMSU7AiJiIio3ClWgNm3bx9CQkJw5MgR7Nq1C0+ePIGPjw+ysrJ0bcLCwhAfH4+4uDjs27cPaWlp6NWrl257Xl4e/Pz8kJOTg8OHDyM2NhYxMTGYMmWKrs2NGzfg5+eHdu3aITk5GWPGjMGQIUOwc+dOCQ6ZiIiI5K5Yt1Hv2LFD73VMTAyqVauGpKQktGnTBpmZmYiKisKaNWvQvn17AEB0dDTc3Nxw5MgRtGzZEr/99hvOnz+P3bt3w8HBAR4eHvjqq68wYcIETJs2DSqVCsuXL4ezszPmzZsHAHBzc8PBgwexYMEC+Pr6SnToREREJFelGgOTmZkJAKhSpQoAICkpCU+ePIG3t7euTYMGDVCrVi0kJiYCABITE9GoUSM4ODjo2vj6+kKr1eLcuXO6Nk/3UdCmoI/CZGdnQ6vV6i1ERERUPpU4wOTn52PMmDFo1aoVGjZsCADQaDRQqVSws7PTa+vg4ACNRqNr83R4KdhesO1FbbRaLR49elTo/kRERMDW1la3ODk5lfTQiIiI6DVX4gATEhKCs2fP4qeffpJyf0ps0qRJyMzM1C2pqallvUtERERkICV6lEBoaCi2bt2K/fv3o2bNmrr1jo6OyMnJQUZGht5VmPT0dDg6OuraHDt2TK+/gruUnm7z7J1L6enpUKvVsLS0LHSfzM3NYW5uXpLDISIiIpkp1hUYIQRCQ0OxceNG7NmzB87OznrbmzVrBjMzMyQkJOjWXbp0CSkpKfDy8gIAeHl54cyZM7hz546uza5du6BWq+Hu7q5r83QfBW0K+iAiIqKKrVhXYEJCQrBmzRps3rwZNjY2ujErtra2sLS0hK2tLYKDgzF27FhUqVIFarUao0aNgpeXF1q2bAkA8PHxgbu7OwYMGIA5c+ZAo9Fg8uTJCAkJ0V1BGTFiBBYvXozx48cjKCgIe/bswfr167FtW/EfPkdERETlT7ECzLJlywAAbdu21VsfHR2NQYMGAQAWLFgApVIJf39/ZGdnw9fXF0uXLtW1NTExwdatWzFy5Eh4eXnBysoKgYGBCA8P17VxdnbGtm3bEBYWhsjISNSsWRMrV67kLdREMlWSJ18DfPo1ERWtWAFGCPHSNhYWFliyZAmWLFlSZJvatWvj119/fWE/bdu2xcmTJ4uze0RERFRB8FlIREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7pmW9A0REUqszcVuJ3ndzlp/Ee0JEhsIrMERERCQ7DDBEREQkOwwwREREJDsMMERERCQ7DDBEREQkOwwwREREJDvFvo16//79mDt3LpKSknD79m1s3LgRPXr00G0XQmDq1Kn47rvvkJGRgVatWmHZsmWoX7++rs29e/cwatQoxMfHQ6lUwt/fH5GRkbC2tta1OX36NEJCQnD8+HFUrVoVo0aNwvjx40t3tEREEuMt20Rlo9gBJisrC++88w6CgoLQq1ev57bPmTMHixYtQmxsLJydnfHll1/C19cX58+fh4WFBQCgX79+uH37Nnbt2oUnT55g8ODBGDZsGNasWQMA0Gq18PHxgbe3N5YvX44zZ84gKCgIdnZ2GDZsWCkPmYhIvhiYiP5V7ADTuXNndO7cudBtQggsXLgQkydPRvfu3QEAq1evhoODAzZt2oSAgABcuHABO3bswPHjx9G8eXMAwDfffIMuXbrgv//9L2rUqIEff/wROTk5WLVqFVQqFd5++20kJydj/vz5DDBEREQk7Uy8N27cgEajgbe3t26dra0tPD09kZiYiICAACQmJsLOzk4XXgDA29sbSqUSR48eRc+ePZGYmIg2bdpApVLp2vj6+mL27Nm4f/8+Kleu/Fzt7OxsZGdn615rtVopD42IqMLh1R56nUk6iFej0QAAHBwc9NY7ODjotmk0GlSrVk1vu6mpKapUqaLXprA+nq7xrIiICNja2uoWJyen0h8QERERvZbKzV1IkyZNQmZmpm5JTU0t610iIiIiA5E0wDg6OgIA0tPT9danp6frtjk6OuLOnTt623Nzc3Hv3j29NoX18XSNZ5mbm0OtVustREREVD5JGmCcnZ3h6OiIhIQE3TqtVoujR4/Cy8sLAODl5YWMjAwkJSXp2uzZswf5+fnw9PTUtdm/fz+ePHmia7Nr1y64uroWOv6FiIiIKpZiB5iHDx8iOTkZycnJAP4duJucnIyUlBQoFAqMGTMGM2bMwJYtW3DmzBkMHDgQNWrU0M0V4+bmhk6dOmHo0KE4duwYDh06hNDQUAQEBKBGjRoAgL59+0KlUiE4OBjnzp3DunXrEBkZibFjx0p24ERERCRfxb4L6cSJE2jXrp3udUGoCAwMRExMDMaPH4+srCwMGzYMGRkZaN26NXbs2KGbAwYAfvzxR4SGhqJDhw66iewWLVqk225ra4vffvsNISEhaNasGd544w1MmTKFt1ATERERgBIEmLZt20IIUeR2hUKB8PBwhIeHF9mmSpUquknritK4cWMcOHCguLtHREREFUC5uQuJiIiIKg4GGCIiIpIdBhgiIiKSHQYYIiIikh0GGCIiIpIdBhgiIiKSHQYYIiIikp1izwNDRERkCHUmbivR+27O8pN4T0gOeAWGiIiIZIcBhoiIiGSHAYaIiIhkh2NgiIioQirJmBuOt3l98AoMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJjmlZ7wAREVF5V2fithK97+YsP4n3pPzgFRgiIiKSHQYYIiIikp3XOsAsWbIEderUgYWFBTw9PXHs2LGy3iUiIiJ6Dby2Y2DWrVuHsWPHYvny5fD09MTChQvh6+uLS5cuoVq1amW9e0RERK+tkoy5Kel4m7Ia3/PaXoGZP38+hg4disGDB8Pd3R3Lly9HpUqVsGrVqrLeNSIiIipjr+UVmJycHCQlJWHSpEm6dUqlEt7e3khMTCz0PdnZ2cjOzta9zszMBABotdpC2+dn/1OifSuqvxcxZi1j1yvPx1bSejw2aWrJpR6PTZpacqnHY5Om1ovqFawXQry4A/EaunXrlgAgDh8+rLd+3Lhx4t133y30PVOnThUAuHDhwoULFy7lYElNTX1hVngtr8CUxKRJkzB27Fjd6/z8fNy7dw/29vZQKBSv3I9Wq4WTkxNSU1OhVqsNsatlUsvY9XhsrPe61TJ2PR6bPOuV52Mzdr2S1hJC4MGDB6hRo8YL272WAeaNN96AiYkJ0tPT9danp6fD0dGx0PeYm5vD3Nxcb52dnV2J90GtVhvli8nYtYxdj8fGeq9bLWPX47HJs155PjZj1ytJLVtb25e2eS0H8apUKjRr1gwJCQm6dfn5+UhISICXl1cZ7hkRERG9Dl7LKzAAMHbsWAQGBqJ58+Z49913sXDhQmRlZWHw4MFlvWtERERUxl7bAPPxxx/j7t27mDJlCjQaDTw8PLBjxw44ODgYtK65uTmmTp363MdRcq9l7Ho8NtZ73WoZux6PTZ71yvOxGbueoWsphHjZfUpEREREr5fXcgwMERER0YswwBAREZHsMMAQERGR7DDAEBERkewwwBAREZHsMMCUUykpKS9/EJaMZWdnIysrq6x3g17R+fPnX9rmhx9+kKze2bNnJevrdfQ6fX8LIXDnzp2y3g16xpdffonc3Nwit6ekpKBjx45G3CPp8TbqMpKVlYV169bh0aNH8PHxQf369SXt38TEBLdv30a1atUk7bes3b17FwMHDsTu3buRn5+PFi1a4IcffkC9evUMVrO8/l8ak6WlJb766it8+umnzz2bLD09HUOHDsXvv/+OBw8eSFJPqVSiRYsWGDJkCAICAmBjYyNJv0VZtGjRS9uYmprC0dERrVu3LvXXkjG/JitVqoQ///wTVatWBQD4+flh5cqVqF69OoB/z1+NGjWQl5dn8H2RmrHPW/v27V+p3Z49e0pVBwBq1aoFe3t7fP/992jYsKHethUrVmDcuHFo1aoVtm/fXupaZaVCB5jVq1e/UruBAweWqk5KSgoGDBiAP/74Ay1btkRUVBQ6duyIK1euAPj3h/v27dvRpk2bUtV5mlKphEajMdov3f37979Su9IeY1BQELZv345PPvkEFhYWWLFiBapXr47ff/+9VP2+iLH/L42lcuXKL33QacEP744dO+LLL78s8fPFfvnlF4wcORKurq6IiYnBW2+9BeDfqy6jR4/G22+/jVWrVkkWRA8cOIDo6Gj8/PPPyM/Ph7+/P4YMGYL3339fkv6f5ezs/NI2+fn5+Pvvv5Gfn48ffvgBvXr1KnE9Y35NPlvLxsYGp06dQt26dQH8G2CqV6+O/Px8SeqlpKS8UrtatWqVulZZnLfatWvDz88PZmZmRbZbsGBBiWsU0Gq1CA0Nxfr16zF16lRMmDABf/31F4KCgnD8+HHMnTsXw4YNK3WdAk8/TLkoBT9POnTogHfeeafUNSt0gKlcuXKR2xQKBbKyspCbm1vqvyx69+6N1NRU3RfT5cuX8dZbbyEqKgpKpRIjR47EvXv3JEndBZRKJdLT03V/NRmaUln0p5EFvyQVCsULL2m+CicnJ6xcuRK+vr4AgCtXrsDNzQ1ZWVkGm+3RmL8snJ2dXxoqFAoFrl27VupasbGxL22Tn5+PO3fuIDo6Gk2aNMHatWtLXO/OnTsYPnw4du3ahWnTpuHAgQPYtWsXZsyYgbCwsGI9Nf5VZWVlYf369YiJicGBAwdQr149BAcHIzAwsMgHwxpSfn4+Zs2ahe+//x4XLlwocT/G/P5+lQAj5RUYExMT3b8Lfj09/bUhhIBCoTDqFR+pztvcuXMRHR2Nv//+G/369UNQUNBzV0ektnnzZgwfPhyOjo64ceMG3n33XaxcuRK1a9eWtE7btm1f+j1c8PPk8uXL+Oabb/Cf//ynVDUrdIApyu3btzF9+nSsWrUK7du3x44dO0rVn6OjI7Zs2YJ3330X9+7dwxtvvIFDhw7pHkx56tQpdOjQAf/73/+k2H0A//7QGTZsGCpVqvTCdvPnz5ekXmZmZqHr//nnH0RGRmLRokWoW7duqccmmJiY4NatW3q/fKysrHDu3DnUqVOnVH0XRalUYsaMGbC2tn5hu08++aTUtSIjI4vcdvPmTaxYsQLZ2dlGv1z/xx9/oGPHjvj7779L3Ve/fv2wdu1aWFlZ4fDhw2jUqJEEe/hyV69eRXR0NL7//ntoNBp06tQJW7ZsMUrtp926dQseHh64e/duifsw5ve3sQOMqakpatasiUGDBqFbt24wNS38iTdS/AVfHFKctwKJiYlYtWoV1q9fD1dXVwQFBaFv374GeTp0eno6+vfvj4SEBFhZWWHr1q344IMPJK9THLGxsQgPDy/1H2IMME958OABZs+ejcjISLz99tuIiIhAu3btSt2vUqnE7du3dc9xsra2xunTpw32A6CgppeXF1QqVZFtFAqFpFd9npafn49Vq1Zh+vTpUCqVmDZtGgIDA194peZVmJiYQKPR6P3lqVarcerUqVe6HFwSSqUSNWvW1PvL8FkKhQLXr183SP179+7hq6++wrJly+Dp6YnZs2ejZcuWkvUvhEBSUhJu3rwJhUIBZ2dnNGnSRO+vqUePHuHbb7/F6NGjS1zn/v37CAkJwebNmzF69GisW7cOlpaWWL16NZo2bSrFobxUVlYWfvzxR0yaNAkZGRmSfc8Z6+PoAsb8/n72e+7Z7zepf35pNBrExsYiOjoaGRkZ6N+/P4KDg+Hm5iZJ/08z9nl71j///IO4uDgsWbIE58+fR1pamqQhZu3atQgNDYWHhweWLl2KqKgoREZG4j//+Q8iIiJgYWEhWa26devi+PHjsLe3f2nbu3fvolOnTkhKSipdUUEiJydHzJs3T9jb2wsXFxcRFxcnaf8KhUKkp6frXltbW4tr167pXms0GqFUKg1a05h++eUX4erqKqpUqSLmzp0rHj9+LFnfCoVC2NnZicqVK+sWhUIhbG1t9dZJqaz+L//55x8xY8YMYWdnJ9555x2xbds2yWvs2bNHODs7C6VSKRQKhVAoFEKpVIq33npL7Nu3T7I68fHxwtHRUbz77rviwoULQgghHj58KEaMGCFUKpWYPHmyePLkiWT1nrVv3z4RGBgorK2thVqtFkOGDBGJiYmS9W9nZ1fkUrlyZaFSqST9Hjfm1+Sz33PPfr/Z2dlJ/vOrwIEDB0RQUJCwsbERnp6e4ttvvxV5eXmS9W/s8/asAwcOiMGDBwtra2vh6ekp/vnnH8n67tWrl7CyshKLFi3SW3/o0CHh4uIiXFxcxOHDhyWrVxY/J1/bp1EbgxACq1evxpQpU5Cbm4uZM2ciODj4hX9pl9SUKVN0l3tzcnLw9ddfw9bWFsC/KVxqhhhP8DL79u3DhAkTcObMGYwePRoTJkzQHaNUoqOjJe3vVRj7/zIvLw/fffcdpk+fDgsLCyxatAj9+/eXfD+uXr2Krl27wtPTEwsWLECDBg0ghMD58+exaNEidOnSRe9KYWn4+/tj6tSpmDhxou4qnJWVFZYtW4ZevXphyJAhiI+PR3JycqlrFUhLS0NMTAxiYmJw9epVvPfee1i0aBF69+4NKysryeoA/15dKszTH0dLecuqMb8my+J7rkDr1q3RunVrzJw5E3369MGIESPg7++PKlWqSNK/sc8boP91qdVq0b9/fxw9ehTu7u6S1tFoNDh58uRzd7i+9957SE5OxsSJE/HBBx8gJydH0rrGVKE/QmrUqBGuX7+OUaNGYcyYMUV+nlzaS3qvMrgJgKR30hj7zpkuXbpg9+7dCAoKwrRp08pkgKShGPP/cv369Zg8eTIyMjLwxRdfYOTIkS/8mKA0QkNDceHCBSQkJDy3TQgBb29vuLu745tvvil1rdOnT6Nx48ZFbtdqtQgLC0NUVFSpawFA586dsXv3brzxxhsYOHAggoKC4OrqKknfr8JQH0cXeJWvyUePHsHS0lKymmXh8OHDWLVqFeLi4nRjRYYNG1bqj6KLYujz1qVLF/z+++/w8fFBUFAQ/Pz8ihzjU1r5+fkv/X/av3+/ZHe/KpVKxMbGvvSP1g8//FCSekAFDzBPn9zCAoYog9HuUomNjUVAQIDB7sx5llKphKmpKaysrF4Y1u7duydJPfEK4zakMn36dIwbN+6lAyaloFQqYWlpiT59+rwwOEsxOLNhw4aIiIhAt27dCt0eHx+PSZMmyXJSuA8//BDBwcHo2rWrQa6oFuXJkyf45ptvMHPmTNjb2+Prr7/GRx99JHmdF31/Z2dnY/HixZg7dy40Gk2pa92/fx8//PADAgMDn/uazMzMxOrVqwvdVlK3b9/G6tWrER0djfv37xvlbh1jnTelUonq1aujWrVqL/xZ9ccff0he29BeJVRK/fu0Qn+EZMi5Q4rrxIkTaN68uWT9NWzYEJ06dcLmzZsL/aHTo0cPLFy4ULKR/Ma8zPz7778jODgYf/75p95tls7Ozli1apWk8+kA/87T8/QPm0OHDqF58+a6Xx4PHjzAhAkTsHTp0lLXatOmzUtvk5YqpKWkpLzwDqCGDRvizz//lKRWUROG2drawsXFRXdHnlQK7i4SQuDEiRMGD7rG/DgaAAICAjBt2jTs2rULKpUK48ePR48ePRAdHY0vvvgCJiYmCAsLk6TW4sWLcfr0aYwaNeq5bba2tjhw4AC0Wi2++OILSerVqlULb775JgIDA/Hhhx/CzMwM+fn5OH36tF67F13Re1XGPm9Tpkwpk4/3jcXo82UZdcRNBffgwYPnBmmdPHlSdO3aVfKBYn379hXh4eFFbv/6669Fv379JK1pDFeuXBGVKlUS7dq1E5s2bRIXL14UFy5cEL/88ov44IMPhJWVld4AaSkolUq9wWk2NjYGH4RtDC8bdCflcdWpU6fQxc7OTigUCtGqVSvx999/S1KrgLEGKAshRMOGDUWlSpXEhAkTxO3bt0VmZmahi1TGjx8vbG1thb+/v6hevbowNTUVQ4cOFY0aNRJr164Vubm5ktV65513xO7du4vcvnv3buHh4SFZvYJzVXC+nj5/T6+XgrHPW35+vmR9vW6e/TlpDBX6I6RnE30BW1tb1KpVS7KknJqait69e+PYsWMwMTFBaGgoZsyYgREjRmDdunXo2bMnwsLC4OnpKUk9AHjrrbewcePGIv9KOXPmDLp37y7prb/r1q3Dli1bkJOTgw4dOmDEiBGS9V3AmOM2Chh7HoynFcwN9MYbb0jet1KpxJ49e4ocEPm///0PHTt2NPhHqNevX0f//v11t3pK4erVq3jnnXfg6emJ0aNHPzdA+cSJE5INUAaM/3F03bp1sXDhQnz44Yc4e/YsGjdujEGDBiEqKkryv/BtbGxw7ty5Ime+TUlJQcOGDaHVaiWp96pX/aSYiM3Y5+29997D6tWrDfrok7JSFjOWV+iPkDw8PKBQKJ57KJpCoYCFhQXGjBmD8PDwUl9OHDduHB4/fozIyEhs2LABkZGROHDgADw9PXHt2jXUrFmzVP0X5tatWy98/ou1tTVu374tWb1ly5YhJCQE9evXh6WlJTZs2IBr165h7ty5ktUAgL179yIiIqLQbQqFAmPGjMGkSZMkrWlsBQN4161bp7tLonLlyggICMCMGTNKPJ1/YTp06PDChwIa43J33bp1MWvWLAQFBUnW58KFC9GyZcvngm6DBg3Qs2dPeHt7Y8GCBZIFXWN/HP3XX3+hWbNmAP79qM/c3NxgsxmbmJggLS2tyACTlpYm6aBaqWeIfRFjn7eaNWvCw8MDs2fPRkhIiFFrG1pgYKDRB41X6ABz48aNQtdnZGQgKSkJX375JSpXrozPPvusVHX279+PDRs2oGXLlujduzccHR3Rr18/jBkzplT9vkjVqlVx6dKlIid3u3jxoqR/1S9evBhTp07F1KlTAfz7jJvhw4dLHmCMOW6jLNy7dw9eXl64desW+vXrp5u86/z584iJiUFCQgIOHz78wsdgvKqivv6fJtXDFV+mVq1akgw4LWDsoGvsmU3z8vL07k4zNTV96UzRJdWkSRNs2rSpyMkTN27ciCZNmkhWr6jZkQvGSxU8RFIKxj5v69evR1xcHEJDQ7Fp0yZER0cb5A/YslAWt9tX6ABTVNKvXbs23nnnHajVakyfPr3UASY9PV0XJKpVq4ZKlSqhc+fOperzZby9vfH111+jU6dOz20TQuDrr7+Gt7e3ZPWuX7+OwMBA3eu+ffsiODgYt2/flvQHzsOHD194N1ClSpUMMq/OypUrdb8gcnNzERMTowuAUv6SDw8Ph0qlwrVr13QzNz+9zcfHB+Hh4ZI87K2or/8HDx5g7dq1iIqKwokTJ4xyF96ZM2ck/cu7rILurVu38Msvv+Dy5csAAFdXV/Tq1QtvvvmmpHWEEBg0aJBuIPnjx48xYsSI5+a32bBhQ6lrhYaGIiAgADVr1sTIkSN1V6Tz8vKwdOlSLFiwAGvWrCl1nQI9evQocptCoUBAQAC+++47Se8KNNZ5A4D/+7//Q9u2bRESEoJGjRphwIABz91KLdUjXoxJqVS+0nPcSvs8PL3+KvIYmJe5ceMGGjVqhIcPH5aqn5dNxW0I165dQ7NmzeDq6opPP/1UNwfGxYsXMW/ePFy+fBknTpyQ7LPYwh4u9+xYEanqGHvcRp06dV7p0vyrXNF4lVorVqzQPazyWTt27MCIESNw8+bNUtd61v79+xEVFYVffvkFNWrUQK9eveDv748WLVqUuu+ixkdkZmYiKSkJn376KQIDAzFlypRS1wJe/nm8IcYtLV26FGPHjkVOTo7uzj+tVguVSoX58+eX+sF1Txs8ePArtZPqr+IvvvgCERERsLGx0X0/X79+HQ8fPsS4ceMwa9YsSeq8SMHXSkhICHr27ImZM2dK0q8xz1uBvLw8hIeHY+bMmWjZsqVegDHkI14MadOmTUX+nExMTMSiRYuQn5+Px48fS1fUqEOGZebw4cOiTp06pe7nZVNxG2L6eyGEOH78uHj77befG83/9ttvi2PHjklaS6FQiOHDh4uwsDDdolKpRFBQkN46KeoUdleCIe5QKAsqlUqkpqYWuT01NVWYm5tLVu/27dsiIiJC1KtXT1SrVk2EhoYKU1NTce7cOclqCCH0vgafXUxMTMTw4cNFdna2pPV+//13cerUqUKXhIQESb9Otm7dKkxMTMSnn34q0tLSdOvT0tJEWFiYMDU1NcijIIzp6NGj4pNPPhFdunQRnTt3FqNHjxZHjx41+n5s375duLq6StJXWZy3s2fPiqZNm4o6deqIPXv2SNr36+bixYuiR48ewsTERAwcOFDcvHlT0v55BaYId+/eRZ8+fVCrVi2sWrWqVH3Fxsa+UrunP4KRUnJyMq5cuQIhBFxcXODh4SF5jVeZbViKvyxe5bL/gwcPJJ/0Kj8/HzExMdiwYYNuTpG6devC398fAwYMkGzw5Jtvvol169ahdevWhW4/cOAAPv74Y6SlpZW6Vrdu3bB//374+fmhX79+6NSpE0xMTGBmZoZTp05JOrX5vn37Cl2vVqtRv359WFtb4+zZs5Kdt4LL2S/68Sbl3SVt27ZF69atMWPGjEK3T548GQcPHsTevXslqfciQgjs2LEDUVFR+Pnnnw1eLyMjAz/88ANCQ0MNXgv496nsDRs2LPWVccD45y0iIgLTp09H3759ERkZ+cIbLeQsLS0NU6dORWxsLHx9fREREWGYiQgljUMy4+HhIZo0afLcUrduXaFSqUSjRo3E7du3y3o36RVotVqxYsUK8e6770p+BSY/P1906dJFKBQK4eHhIQICAsTHH38sGjduLBQKhejevbtktQYPHizatGlT6NWIx48fiw8++EAMHjxYklomJiYiLCxMXL58WW+9Ia7AFMVQ5+3mzZsvXc6cOSNZPRsbG3Hx4sUit1+8eFHY2NhIVq8w169fF5MnTxY1a9YU5ubmws/Pz6D1du/eLfr06SMsLCxElSpVDFrraQkJCaJ+/fqS9GXs8+bo6Cji4+Ml6+91k5GRIcaPHy8sLS2Fl5eX2L9/v0HrVehBvEUNFlOr1XB1dYWvr68kMzIeO3YMzZo1K7Kv7OxsbN68Gb179y51rQLh4eGv1E6qMQfPMuT8JU8rbNzG4sWLJa0RExODAwcOICEh4bnnouzZswc9evTA6tWrMXDgwFLXCg8PR/PmzVG/fn2EhITo5i+5cOECli5diuzsbHz//felrgMABw8eRFRUFJo1awY3NzcMGDAAAQEBkvT9MoY+b8YeoJyXlwczM7Mit5uZmRlkMHR2djZ+/vlnREVF4eDBg8jLy8N///tfBAcHSza1/9NSU1MRHR2N6OhopKSkICAgABs3bkSHDh0kr1WY5ORkfPbZZ/Dz85OkP2Oft3Xr1j03Hmz16tWYOnUqsrKy0KNHD3zzzTdGewSMlObMmYPZs2fD0dERa9euRffu3Q1f1KDxiIQQZTOTq4eHR5FLkyZNRKVKlSSvef/+ffGf//xH2Nvb68Y32Nvbi5CQEHH//n3J6hhr3EaBjh07ioiIiCK3f/3118LHx0eyetevXxedOnV6bgZZX19fceXKFcnqFHj48KGIiooSrVq1EmZmZkKpVIqFCxcKrVYraR1jn7en7du3TwwcOFBYWVmJ+vXriwkTJkg6DqxFixZi/vz5RW6fN2+eaNGihWT1Tpw4IUaOHCns7OxE8+bNRWRkpNBoNAb5/8zJyRHr168XPj4+wtLSUvTs2VPExcUZ7Nw9PV7w6UWlUum+D6SaHdfY561Tp05i1qxZutenT58WpqamYsiQIWLevHnC0dFRTJ06VbJ6xqRQKESlSpXEhx9+KHr27FnkIqUKfQWmwKNHj7Br1y69W+i8vb0lm5RHPPM5/LOvi1pXGidPnix0fcFj1M+ePYuhQ4dKVs9Y85c8PW5j4cKFunEby5cvl+IwCnX69GnMmTOnyO2dO3cu8lk/JeHs7Izt27fj/v37uHLlCgCgXr16Rd55VVpWVlYICgpCUFAQLl26hKioKMyaNQsTJ05Ex44di5yXozjK4rxpNBrExMQgKioKWq0WvXv3RnZ2NjZt2iTp+B4ACAkJwciRI2Fubo5hw4bp7irJzc3FihUrMHnyZMlmGQYAT09PjBo1CkeOHDH4U7bffPNNNGjQAP3798dPP/2k+x7u06ePQeotWLCg0DFlBVfGpTx3xj5vycnJ+Oqrr3Svf/rpJ3h6euK7774DADg5OWHq1KmYNm2aZDWNZeDAgcZ/zpOkcUiGNm/eLKpWrfrcnSxVq1YVW7ZskaTGs8+csba2NvqzdK5fvy769esnTE1NRe/evZ8b91Bao0ePFg0bNhQajea5bbdv3xaNGjUSY8aMKXWdshi3YWZmpneHwrNu3bolVCqVQWoXJi4uzuA1cnNzxcaNG0W3bt0k6c/Y561r165CrVaLPn36iK1bt+qeDWTIr5NPP/1UKBQKoVarRZMmTYSHh4dQq9VCqVRK8rX/NB8fH2FjYyP69u0rtm/frnvGjiGOr3LlyqJNmzbi22+/1bvyYcyxUoZkzPNmbm4uUlJSdK9btWolZsyYoXt948YNYW1tLWnN8qxCB5hDhw4JMzMz4e/vLw4fPizu378v7t+/Lw4dOiR69eolVCqVSExMLHWdsgwwd+/eFaGhoUKlUon27dtLfvt0gdq1a4sdO3YUuX379u2idu3apa6TmJgohgwZImxsbMS7774rvvnmG3H37l2D/jBVKpXizp07RW6X+vw9efJEnDlzRly6dElv/aZNm0Tjxo2NGpakYuzzVlYDlBMTE8Unn3wiOnfurLvVWIqfIYVJSUkR06ZNE3Xq1BEODg7ik08+EaampuL8+fOS1nn06JH44YcfRLt27YSlpaXo1auX2LBhgzAzMzPI/+WLbrl/+tZ7KRnrvNWqVUv3INHs7GxhaWmp96DM06dPG2RKDWN40cdGBUuvXr0krVmhb6Pu0qULnJycsGLFikK3Dx8+HKmpqfj1119LVefZydfee+89rF+/XjeFtCEmX8vKysJ///tfzJ8/H/Xq1UNERAR8fHwk6/9Z5ubmL3yu019//YV69epJNolRVlYW1q1bh1WrVuHYsWPIy8vD/PnzERQUJPmtiUqlEp07dy5yYF12djZ27Nghyfk7e/YsunbtitTUVABA9+7dsWzZMvTu3Vv3sV9oaKhspx831nk7cuQIoqKisG7dOr0BytWrV5f8FvHXwe7du7Fq1Sps3LgRTk5O+Oijj/DRRx+hadOmkta5du0aVq1ahdWrV+PWrVvo06cPBg0ahPbt20tywwNQRhOiGcnIkSNx6tQpzJ49G5s2bUJsbCzS0tJ0j4X48ccfsXDhQhw/fryM97T4jD25IoCK/RFS5cqVxenTp4vcfurUKWFnZ1fqOi+bdM0Qk685ODjoHhOfnJxc5IReUqlRo4Y4cOBAkdv3798vqlevLlm9p128eFGMGzdOODo6CgsLC8k+9igwaNCgV1qk0KVLF9GhQwcRHx8v+vbtKxQKhWjQoIGYO3eu+OeffySp8bow9HkTwngDlC9fviwCAgIKHVyakZEh+vTpo3fVtbRyc3PFrFmzxHvvvSeaN28uJkyYIP755x9x7949sWjRIuHh4WHQj6Xz8vLEr7/+Kvz9/YVKpRL29vYGqyWE4SZEM/Z5u3v3rnj//feFQqEQNjY2YsOGDXrb27dvLz7//HPJ6pV3FTrAWFhYvPAb4ebNm8LCwqLUdU6fPv1K81JIqbCQZMjQZMz5S4oi9biNslC1alVx8uRJIcS/P0AVCoVYvXp12e6UgRnrvBkyMA0dOlSMGzeuyO3jx48XI0aMkKxeeHi4UCqVwsfHR3Tv3l1YWFg89/2VlJQkWb3ExETx+eefi88++0xs375db9udO3fEvHnzJKv1tFu3bokhQ4YIMzMz0bVrV0nn7hHC+OetQEZGhm5c1tP+/vtvSWekLu8qdIBp1KiRWLVqVZHbo6KiRKNGjUpdR6FQCE9PT/Htt99K/pdfUV4lMEkZmlJTU4WDg4OoVauWmD17tti8ebPYtGmTiIiIEE5OTqJatWp6g9eocIWNl5J6wHVFZ4jA5OLi8sLxZSdOnBAuLi6S1atXr55Yvny57vWuXbuESqUSeXl5ktUoEBcXJ5RKpbCyshJ2dnZCqVSKuXPnSl7nacaaEM3Y542kVaEDzPz580WVKlUKfdbF1q1bhb29vSR/Wezfv18MHjxY2NjYCCsrKzFw4ECDz1BYFow9f0l5pFQqxdWrV0VmZqbIyMgQNjY24tSpUyIzM1NvodfLq1zNtbS0lKyeSqV67g8Cc3PzFz5Hq6SaNm0qhg8frrtiMHPmTIMONJ09e7aoUqWKcHd3F5s2bTJYHSGMf95IWhV6EG9+fj4+/vhj/PLLL3B1dYWbm5tu1tMrV66gR48eiIuLg1KplKReVlYW1q9fr5vZtV69eggODkZgYCAcHR0lqVGgqLk7bG1t4eLigurVq0ta72nGmr+kPHr2kfRCiEJfG2JWVyo5R0dHrFmzBu3bty90e0JCAvr16weNRiNJvWefcA/8+/T306dPS/6Ue2trayQnJ+ueXJ+TkwMrKyvcunWryKd9l4ZSqYSlpSW8vb1fODB4w4YNpa5l7PNG0qrQAabAunXrsGbNGt0vXRcXFwQEBBh0WvWrV68iOjoa33//PTQaDTp16iTJhGEFXhS6FAoFAgIC8N1336FSpUqS1XyZn3/+GR999JHR6slRUQ89fNYHH3xg4D2h4ujduzeePHmCjRs3Frq9e/fuUKlUiIuLk6ReYXfGxcfHo3379rCystKtk+KXvFKphEaj0QsrNjY2OHXqFOrWrVvq/p81aNCgV5oQTYq7WYx93khaDDBlKCsrCz/++CMmTZqEjIwMo/xVnZmZiaSkJISEhKBnz56YOXOmZH3n5ubi4sWLUKlUcHFx0a3fvHkzpkyZgosXLyI7O1uyeuVRwbNstmzZgpycHHTo0AFTp06VbFZoMoyTJ0/Cy8sLXbt2xfjx43Wz4168eBFz5szBtm3bcPjwYcluazbmLatKpRIzZsyAtbW1bt2ECRMwbtw4vWedffLJJ6WuZWzGPm8ksTL78Oo1UBYTJgnx73NZAgMDhbW1tVCr1WLIkCEGm+yqKNu3bxeurq6S9XfmzBlRu3Zt3f9bz549hUajEW3atBFVqlQREyZMMMjn8+XNq9xdQq+n+Ph4UbVq1ed+hlStWlVs3ry5rHevxGrXri3q1KnzwsXZ2bmsd7PECjtvBbOxy/m8VQQV+grM5s2bi9wm9YRJaWlpiImJQUxMDK5evYr33nsPwcHB6N27t94lX2O5efMmGjZsiIcPH0rSn5+fH7KzszFmzBisXbsWa9euhaurK4KDgxESEsIrCK+ofv36+OyzzzB8+HAA/05Q5ufnh0ePHkk2FosM59GjR9ixYweuXr0KIQRcXFzg4+Nj1I9qqfiePW+urq7w8fHhz63XXdnmp9ePISZM6tSpkzA1NRWOjo5i/Pjx4uLFixLsaekkJCSI+vXrS9ZfRZy/xBCMeXcJSSchIUG4ubkVOSGau7u7bO88PHz4sIiPj9dbFxsbK+rUqSOqVq0qhg4dKh4/flxGe1c6hR1bTExMuTi2ioB/0v0/aWlpGDp0KBo1aoTc3FwkJycjNjYWtWvXLnXfZmZm+Pnnn/HXX39h9uzZBn967MskJyfjs88+g5+fn2R9/u9//0ONGjUA/Hunk5WVFVq2bClZ/xVFbm4uLCws9NaZmZnhyZMnZbRH9CoWLlyIoUOHQq1WP7fN1tYWw4cPx/z588tgz0pv+vTpOHfunO71mTNnEBwcDG9vb0ycOBHx8fGIiIgowz0sufDw8OeObejQoeXi2CqCCv0REvDvoNaZM2fim2++gYeHB2bPno3333+/rHer1CpXrlzoSP6srCzk5uaiY8eOWL9+faE/cEvCxMQEly9fRtWqVSGEgJOTEw4ePIg6derotZOqXnllzLtLSDq1a9fGjh074ObmVuj2ixcvwsfHBykpKUbes9KrXr064uPj0bx5cwDAF198gX379uHgwYMAgLi4OEydOhXnz58vy90skfJ8bBWBaVnvQFmaM2cOZs+eDUdHR6xduxbdu3cv612SzIIFCwoNMGq1Gq6urpI/zE78v8/7n37dpEkTvdecv+TlAgMDn1vXv3//MtgTKo709HSYmZkVud3U1BR379414h5J5/79+3BwcNC93rdvHzp37qx73aJFC93DR+WmPB9bRVChA8zEiRNhaWmJevXqITY2FrGxsYW2k+NfuwMGDDDq7bi///67QfqtaCR9UisZzZtvvomzZ8/qJnt71unTpw06eaQhOTg44MaNG3ByckJOTg7++OMPTJ8+Xbf9wYMHLwxvr7PyfGwVQYUOMAMHDnylCZPkaObMmZg2bRq8vb1haWmJyMhI3LlzB6tWrTJIvdatW3P+EqqwunTpgi+//BKdOnV6bgzTo0ePMHXqVHTt2rWM9q50unTpgokTJ2L27NnYtGkTKlWqpPcx++nTp/HWW2+V4R6WXHk+tgqhLEcQk+EY82FvQnD+EqrYNBqNqFGjhnBychKzZ88WmzZtEps2bRKzZs0STk5OokaNGkKj0ZT1bpbI3bt3xfvvvy8UCoWwsbERGzZs0Nvevn178fnnn5fR3pVOeT62iqDCD+Itr8zNzXH16lU4OTnp1llYWODq1auoWbOm5PU4fwlVdH/++SdGjhyJnTt3ouDHqkKhgK+vL5YsWSL5M4qMLTMzE9bW1s89n+jevXuwtraGSqUqoz0rvfJ8bOUZA0w5ZcyHvQHGD0xEr6v79+/rJkSrX78+KleuXNa7RFQuVegxMOWZEAKDBg3Sux338ePHGDFihEFux+X8JUT/qly5Mlq0aFHWu0FU7jHAlFPGvh3X2IGJiIgqNn6ERJIw5tNxiYiIGGCIiIhIdnh7CBEREckOAwwRERHJDgMMERERyQ4DDBEREckOAwwRERHJDgMMEZW5QYMGQaFQQKFQQKVSoV69eggPD0dubi4AIC8vDwsWLECjRo1gYWGBypUro3Pnzjh06JBeP3l5eZg1axYaNGgAS0tLVKlSBZ6enli5cmVZHBYRGRAnsiOi10KnTp0QHR2N7Oxs/PrrrwgJCYGZmRkmTpyIgIAA7N69G3PnzkWHDh2g1WqxZMkStG3bFnFxcejRowcAYPr06VixYgUWL16M5s2bQ6vV4sSJE7h//37ZHhwRSY7zwBBRmRs0aBAyMjKwadMm3TofHx88ePAAY8aMQUBAALZs2YJu3brpvc/f3x/79u3Dn3/+CSsrK3h4eKBnz56YOnWqkY+AiIyNHyER0WvJ0tISOTk5WLNmDVxcXJ4LLwDw6aef4u+//8auXbsAAI6OjtizZw/u3r1r7N0lIiNjgCGi14oQArt378bOnTvRvn17XL58GW5uboW2LVh/+fJlAMD8+fNx9+5dODo6onHjxhgxYgS2b99utH0nIuNhgCGi18LWrVthbW0NCwsLdO7cGR9//DGmTZsG4N9Q8yrc3d1x9uxZHDlyBEFBQbhz5w66deuGIUOGGHDPiagsMMAQ0WuhXbt2SE5OxpUrV/Do0SPExsbCysoKLi4uuHDhQqHvKVjv4uKiW6dUKtGiRQuMGTMGGzZsQExMDKKionDjxg2jHAcRGQcDDBG9FqysrFCvXj3UqlULpqb//w2SAQEBuHLlCuLj4597z7x582Bvb4+OHTsW2a+7uzsAICsrS/qdJqIyw9uoiei1FhAQgLi4OAQGBj53G/WWLVsQFxcHKysrAMBHH32EVq1a4b333oOjoyNu3LiBSZMmwcXFBQ0aNCjjIyEiKfEKDBG91hQKBdavX4/PP/8cCxYsgKurK95//338+eef2Lt3r24OGADw9fVFfHw8unXrBhcXFwQGBqJBgwb47bff9K7qEJH8cR4YIiIikh1egSEiIiLZYYAhIiIi2WGAISIiItlhgCEiIiLZYYAhIiIi2WGAISIiItlhgCEiIiLZYYAhIiIi2WGAISIiItlhgCEiIiLZYYAhIiIi2fn/AOoNJZ0RpdE/AAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we have tested to find that the tagging and counting works for 2019, repeat the process for the remaining years. Since we are creating new dataframes per year, this is a bit messy (the alterative was to write a helper function that did all this in a for loop, but since we need to create the dataframe variables manually anyway it's currently trivial to copy/paste the code for the five years)."
      ],
      "metadata": {
        "id": "9YAkCtwCA4Nr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# find the counts for the remaining years\n",
        "pos_2020_df = pos_df[pos_df['Year'] == 2020]\n",
        "counts_20 = pos_2020_df['POS'].value_counts()\n",
        "total_20 = counts_20.sum()\n",
        "\n",
        "pos_2021_df = pos_df[pos_df['Year'] == 2021]\n",
        "counts_21 = pos_2021_df['POS'].value_counts()\n",
        "total_21 = counts_21.sum()\n",
        "\n",
        "pos_2022_df = pos_df[pos_df['Year'] == 2022]\n",
        "counts_22 = pos_2022_df['POS'].value_counts()\n",
        "total_22 = counts_22.sum()\n",
        "\n",
        "pos_2023_df = pos_df[pos_df['Year'] == 2023]\n",
        "counts_23 = pos_2023_df['POS'].value_counts()\n",
        "total_23 = counts_23.sum()\n",
        "\n",
        "pos_2024_df = pos_df[pos_df['Year'] == 2024]\n",
        "counts_24 = pos_2024_df['POS'].value_counts()\n",
        "total_24 = counts_24.sum()"
      ],
      "metadata": {
        "id": "lFkuo9RuSfIn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that when doing the counts, we're not looking a proportion across total number of words, we are looking at proportion across total number of **tags** (which include punctuation, spaces, etc.)\n",
        "\n",
        "The proportion is given by a decimal number, calculated by taking `count_of_tag / total_tag_count` for each tag per year. To get the percentage, multiply all values by 100."
      ],
      "metadata": {
        "id": "QLvjWEz4BbVy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# read these counts into their respective dataframes\n",
        "counts_19_df = counts_19.rename_axis('POS').reset_index(name='counts')\n",
        "counts_19_df['proportion'] = counts_19_df['counts'] / total_19\n",
        "\n",
        "counts_20_df = counts_20.rename_axis('POS').reset_index(name='counts')\n",
        "counts_20_df['proportion'] = counts_20_df['counts'] / total_20\n",
        "\n",
        "counts_21_df = counts_21.rename_axis('POS').reset_index(name='counts')\n",
        "counts_21_df['proportion'] = counts_21_df['counts'] / total_21\n",
        "\n",
        "counts_22_df = counts_22.rename_axis('POS').reset_index(name='counts')\n",
        "counts_22_df['proportion'] = counts_22_df['counts'] / total_22\n",
        "\n",
        "counts_23_df = counts_23.rename_axis('POS').reset_index(name='counts')\n",
        "counts_23_df['proportion'] = counts_23_df['counts'] / total_23\n",
        "\n",
        "counts_24_df = counts_24.rename_axis('POS').reset_index(name='counts')\n",
        "counts_24_df['proportion'] = counts_24_df['counts'] / total_24"
      ],
      "metadata": {
        "id": "DUzG4nZWp6fi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "counts_19_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "2eIxrZWCqw8H",
        "outputId": "d46e2ae8-16f6-450f-c436-d9c6b82d48cc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "     POS  counts  proportion\n",
              "0   NOUN    6641    0.177795\n",
              "1   VERB    4168    0.111587\n",
              "2  PUNCT    4099    0.109740\n",
              "3  PROPN    3557    0.095229\n",
              "4    ADP    3551    0.095069"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-85c42827-bc77-4678-b915-b54723cf360f\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>POS</th>\n",
              "      <th>counts</th>\n",
              "      <th>proportion</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>NOUN</td>\n",
              "      <td>6641</td>\n",
              "      <td>0.177795</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>VERB</td>\n",
              "      <td>4168</td>\n",
              "      <td>0.111587</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>PUNCT</td>\n",
              "      <td>4099</td>\n",
              "      <td>0.109740</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>PROPN</td>\n",
              "      <td>3557</td>\n",
              "      <td>0.095229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ADP</td>\n",
              "      <td>3551</td>\n",
              "      <td>0.095069</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-85c42827-bc77-4678-b915-b54723cf360f')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-85c42827-bc77-4678-b915-b54723cf360f button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-85c42827-bc77-4678-b915-b54723cf360f');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-332d94a2-668f-4a4e-a6b6-992072539ae8\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-332d94a2-668f-4a4e-a6b6-992072539ae8')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-332d94a2-668f-4a4e-a6b6-992072539ae8 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "counts_19_df",
              "summary": "{\n  \"name\": \"counts_19_df\",\n  \"rows\": 18,\n  \"fields\": [\n    {\n      \"column\": \"POS\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 18,\n        \"samples\": [\n          \"NOUN\",\n          \"VERB\",\n          \"AUX\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"counts\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1779,\n        \"min\": 52,\n        \"max\": 6641,\n        \"num_unique_values\": 18,\n        \"samples\": [\n          6641,\n          4168,\n          1915\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"proportion\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.04765283818335346,\n        \"min\": 0.00139216106232598,\n        \"max\": 0.1777950310559006,\n        \"num_unique_values\": 18,\n        \"samples\": [\n          0.1777950310559006,\n          0.11158706361105161,\n          0.051269008352966376\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Making summary tables\n",
        "This more easily compares the numbers across the years. Create two summary dataframes: one to compare absolute counts, the other to compare proportions. The row labels are the years, the column labels are the parts of speech tags in alphabetical order, taken from [this documentation](https://melaniewalsh.github.io/Intro-Cultural-Analytics/05-Text-Analysis/13-POS-Keywords.html)."
      ],
      "metadata": {
        "id": "t0VKskI3rUBl"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# helper function to produce a row of all the tag counts from a year\n",
        "# where df is the count_xx_df produced in the previous section\n",
        "def summarize_counts(df):\n",
        "  tags = ['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM',\n",
        "          'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X', 'SPACE']\n",
        "\n",
        "  count = []\n",
        "\n",
        "  for tag in tags:\n",
        "    if tag in df['POS'].values:\n",
        "      count.append(df.loc[df['POS'] == tag, 'counts'].item())\n",
        "\n",
        "    else:\n",
        "      count.append(0)\n",
        "\n",
        "  return count\n",
        "\n",
        "\n",
        "# helper function to produce a row of all the tag proportions from a year\n",
        "def summarize_prop(df):\n",
        "  tags = ['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM',\n",
        "          'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X', 'SPACE']\n",
        "\n",
        "  prop = []\n",
        "\n",
        "  for tag in tags:\n",
        "    if tag in df['POS'].values:\n",
        "      prop.append(df.loc[df['POS'] == tag, 'proportion'].item())\n",
        "\n",
        "    else:\n",
        "      prop.append(0)\n",
        "\n",
        "  return prop"
      ],
      "metadata": {
        "id": "yuOmjElmrYc8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# create summary dataframes\n",
        "counts_summary = []\n",
        "prop_summary = []\n",
        "\n",
        "counts_summary.append(summarize_counts(counts_19_df))\n",
        "counts_summary.append(summarize_counts(counts_20_df))\n",
        "counts_summary.append(summarize_counts(counts_21_df))\n",
        "counts_summary.append(summarize_counts(counts_22_df))\n",
        "counts_summary.append(summarize_counts(counts_23_df))\n",
        "counts_summary.append(summarize_counts(counts_24_df))\n",
        "\n",
        "prop_summary.append(summarize_prop(counts_19_df))\n",
        "prop_summary.append(summarize_prop(counts_20_df))\n",
        "prop_summary.append(summarize_prop(counts_21_df))\n",
        "prop_summary.append(summarize_prop(counts_22_df))\n",
        "prop_summary.append(summarize_prop(counts_23_df))\n",
        "prop_summary.append(summarize_prop(counts_24_df))\n",
        "\n",
        "# create dataframes\n",
        "counts_summary_df = pd.DataFrame(counts_summary)\n",
        "prop_summary_df = pd.DataFrame(prop_summary)\n",
        "\n",
        "# row and column headers\n",
        "row_labels = ['2019', '2020', '2021', '2022', '2023', '2024']\n",
        "col_labels = ['ADJ', 'ADP', 'ADV', 'AUX', 'CONJ', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM',\n",
        "          'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X', 'SPACE']\n",
        "\n",
        "counts_summary_df.index = row_labels\n",
        "counts_summary_df.columns = col_labels\n",
        "\n",
        "prop_summary_df.index = row_labels\n",
        "prop_summary_df.columns = col_labels"
      ],
      "metadata": {
        "id": "y-w7dNFFr0qo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "counts_summary_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "6BfUgXo3xatX",
        "outputId": "d37dbcbd-3a95-4d0f-ef7d-77c1999199ed"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "       ADJ    ADP   ADV   AUX  CONJ  CCONJ    DET  INTJ   NOUN   NUM  PART  \\\n",
              "2019  2175   3551  1217  1915     0   1085   2708    52   6641   744  1055   \n",
              "2020  7905  12701  4676  7367     0   3505  10212   148  22970  3065  3555   \n",
              "2021  9021  14242  5087  7557     0   4082  11522   126  26692  3497  3982   \n",
              "2022  5619   8722  3281  4374     0   2392   6841    75  16320  2473  2256   \n",
              "2023  7231  11285  3690  5565     0   3139   9163    78  21830  2109  2978   \n",
              "\n",
              "      PRON  PROPN  PUNCT  SCONJ  SYM   VERB    X  SPACE  \n",
              "2019  2502   3557   4099    670  133   4168   60   1020  \n",
              "2020  8740  15448  15684   2669  658  14936  259   4237  \n",
              "2021  8797  14316  17555   2766  453  16357  218   4261  \n",
              "2022  5234   8624  10639   1683  518   9725  212   2585  \n",
              "2023  6552  12022  13904   2090  576  13075  225   3906  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-dca27555-9060-4772-aa11-c7fd19fd4c45\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ADJ</th>\n",
              "      <th>ADP</th>\n",
              "      <th>ADV</th>\n",
              "      <th>AUX</th>\n",
              "      <th>CONJ</th>\n",
              "      <th>CCONJ</th>\n",
              "      <th>DET</th>\n",
              "      <th>INTJ</th>\n",
              "      <th>NOUN</th>\n",
              "      <th>NUM</th>\n",
              "      <th>PART</th>\n",
              "      <th>PRON</th>\n",
              "      <th>PROPN</th>\n",
              "      <th>PUNCT</th>\n",
              "      <th>SCONJ</th>\n",
              "      <th>SYM</th>\n",
              "      <th>VERB</th>\n",
              "      <th>X</th>\n",
              "      <th>SPACE</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019</th>\n",
              "      <td>2175</td>\n",
              "      <td>3551</td>\n",
              "      <td>1217</td>\n",
              "      <td>1915</td>\n",
              "      <td>0</td>\n",
              "      <td>1085</td>\n",
              "      <td>2708</td>\n",
              "      <td>52</td>\n",
              "      <td>6641</td>\n",
              "      <td>744</td>\n",
              "      <td>1055</td>\n",
              "      <td>2502</td>\n",
              "      <td>3557</td>\n",
              "      <td>4099</td>\n",
              "      <td>670</td>\n",
              "      <td>133</td>\n",
              "      <td>4168</td>\n",
              "      <td>60</td>\n",
              "      <td>1020</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2020</th>\n",
              "      <td>7905</td>\n",
              "      <td>12701</td>\n",
              "      <td>4676</td>\n",
              "      <td>7367</td>\n",
              "      <td>0</td>\n",
              "      <td>3505</td>\n",
              "      <td>10212</td>\n",
              "      <td>148</td>\n",
              "      <td>22970</td>\n",
              "      <td>3065</td>\n",
              "      <td>3555</td>\n",
              "      <td>8740</td>\n",
              "      <td>15448</td>\n",
              "      <td>15684</td>\n",
              "      <td>2669</td>\n",
              "      <td>658</td>\n",
              "      <td>14936</td>\n",
              "      <td>259</td>\n",
              "      <td>4237</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2021</th>\n",
              "      <td>9021</td>\n",
              "      <td>14242</td>\n",
              "      <td>5087</td>\n",
              "      <td>7557</td>\n",
              "      <td>0</td>\n",
              "      <td>4082</td>\n",
              "      <td>11522</td>\n",
              "      <td>126</td>\n",
              "      <td>26692</td>\n",
              "      <td>3497</td>\n",
              "      <td>3982</td>\n",
              "      <td>8797</td>\n",
              "      <td>14316</td>\n",
              "      <td>17555</td>\n",
              "      <td>2766</td>\n",
              "      <td>453</td>\n",
              "      <td>16357</td>\n",
              "      <td>218</td>\n",
              "      <td>4261</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2022</th>\n",
              "      <td>5619</td>\n",
              "      <td>8722</td>\n",
              "      <td>3281</td>\n",
              "      <td>4374</td>\n",
              "      <td>0</td>\n",
              "      <td>2392</td>\n",
              "      <td>6841</td>\n",
              "      <td>75</td>\n",
              "      <td>16320</td>\n",
              "      <td>2473</td>\n",
              "      <td>2256</td>\n",
              "      <td>5234</td>\n",
              "      <td>8624</td>\n",
              "      <td>10639</td>\n",
              "      <td>1683</td>\n",
              "      <td>518</td>\n",
              "      <td>9725</td>\n",
              "      <td>212</td>\n",
              "      <td>2585</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023</th>\n",
              "      <td>7231</td>\n",
              "      <td>11285</td>\n",
              "      <td>3690</td>\n",
              "      <td>5565</td>\n",
              "      <td>0</td>\n",
              "      <td>3139</td>\n",
              "      <td>9163</td>\n",
              "      <td>78</td>\n",
              "      <td>21830</td>\n",
              "      <td>2109</td>\n",
              "      <td>2978</td>\n",
              "      <td>6552</td>\n",
              "      <td>12022</td>\n",
              "      <td>13904</td>\n",
              "      <td>2090</td>\n",
              "      <td>576</td>\n",
              "      <td>13075</td>\n",
              "      <td>225</td>\n",
              "      <td>3906</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dca27555-9060-4772-aa11-c7fd19fd4c45')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-dca27555-9060-4772-aa11-c7fd19fd4c45 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-dca27555-9060-4772-aa11-c7fd19fd4c45');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-966d1d7a-b40e-4039-a12a-8efac1b76ff7\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-966d1d7a-b40e-4039-a12a-8efac1b76ff7')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-966d1d7a-b40e-4039-a12a-8efac1b76ff7 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "counts_summary_df",
              "summary": "{\n  \"name\": \"counts_summary_df\",\n  \"rows\": 6,\n  \"fields\": [\n    {\n      \"column\": \"ADJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3098,\n        \"min\": 1525,\n        \"max\": 9021,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          2175,\n          7905,\n          1525\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADP\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 4868,\n        \"min\": 2480,\n        \"max\": 14242,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          3551,\n          12701,\n          2480\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADV\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1737,\n        \"min\": 920,\n        \"max\": 5087,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1217,\n          4676,\n          920\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AUX\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 2655,\n        \"min\": 1329,\n        \"max\": 7557,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1915,\n          7367,\n          1329\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"CONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"CCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1365,\n        \"min\": 660,\n        \"max\": 4082,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1085\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"DET\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3954,\n        \"min\": 2065,\n        \"max\": 11522,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          2708\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"INTJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 42,\n        \"min\": 37,\n        \"max\": 148,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          52\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NOUN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 9103,\n        \"min\": 4542,\n        \"max\": 26692,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          6641\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NUM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1229,\n        \"min\": 456,\n        \"max\": 3497,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          744\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PART\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1333,\n        \"min\": 699,\n        \"max\": 3982,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1055\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PRON\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 3011,\n        \"min\": 1774,\n        \"max\": 8797,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          2502\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PROPN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 5347,\n        \"min\": 3023,\n        \"max\": 15448,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          3557\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PUNCT\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 6011,\n        \"min\": 3272,\n        \"max\": 17555,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          4099\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 980,\n        \"min\": 466,\n        \"max\": 2766,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          670\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SYM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 210,\n        \"min\": 133,\n        \"max\": 658,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          133\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"VERB\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 5581,\n        \"min\": 3088,\n        \"max\": 16357,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          4168\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"X\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 81,\n        \"min\": 60,\n        \"max\": 259,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          60\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SPACE\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1597,\n        \"min\": 771,\n        \"max\": 4261,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1020\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "prop_summary_df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "cwmqzW_ozAaZ",
        "outputId": "b69ab549-759b-4418-d939-a4abf7f27077"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "           ADJ       ADP       ADV       AUX  CONJ     CCONJ       DET  \\\n",
              "2019  0.058230  0.095069  0.032582  0.051269     0  0.029048  0.072499   \n",
              "2020  0.056979  0.091549  0.033705  0.053101     0  0.025264  0.073608   \n",
              "2021  0.059928  0.094612  0.033794  0.050202     0  0.027117  0.076542   \n",
              "2022  0.061361  0.095246  0.035829  0.047765     0  0.026121  0.074705   \n",
              "2023  0.060552  0.094500  0.030900  0.046601     0  0.026286  0.076730   \n",
              "\n",
              "          INTJ      NOUN       NUM      PART      PRON     PROPN     PUNCT  \\\n",
              "2019  0.001392  0.177795  0.019919  0.028245  0.066984  0.095229  0.109740   \n",
              "2020  0.001067  0.165567  0.022092  0.025624  0.062998  0.111349  0.113050   \n",
              "2021  0.000837  0.177319  0.023231  0.026453  0.058440  0.095103  0.116620   \n",
              "2022  0.000819  0.178218  0.027006  0.024636  0.057157  0.094176  0.116181   \n",
              "2023  0.000653  0.182803  0.017661  0.024938  0.054866  0.100672  0.116431   \n",
              "\n",
              "         SCONJ       SYM      VERB         X     SPACE  \n",
              "2019  0.017937  0.003561  0.111587  0.001606  0.027308  \n",
              "2020  0.019238  0.004743  0.107658  0.001867  0.030540  \n",
              "2021  0.018375  0.003009  0.108662  0.001448  0.028306  \n",
              "2022  0.018379  0.005657  0.106199  0.002315  0.028229  \n",
              "2023  0.017502  0.004823  0.109489  0.001884  0.032709  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-c959abdc-ebb6-4851-b69a-f1f2fa973f2b\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>ADJ</th>\n",
              "      <th>ADP</th>\n",
              "      <th>ADV</th>\n",
              "      <th>AUX</th>\n",
              "      <th>CONJ</th>\n",
              "      <th>CCONJ</th>\n",
              "      <th>DET</th>\n",
              "      <th>INTJ</th>\n",
              "      <th>NOUN</th>\n",
              "      <th>NUM</th>\n",
              "      <th>PART</th>\n",
              "      <th>PRON</th>\n",
              "      <th>PROPN</th>\n",
              "      <th>PUNCT</th>\n",
              "      <th>SCONJ</th>\n",
              "      <th>SYM</th>\n",
              "      <th>VERB</th>\n",
              "      <th>X</th>\n",
              "      <th>SPACE</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2019</th>\n",
              "      <td>0.058230</td>\n",
              "      <td>0.095069</td>\n",
              "      <td>0.032582</td>\n",
              "      <td>0.051269</td>\n",
              "      <td>0</td>\n",
              "      <td>0.029048</td>\n",
              "      <td>0.072499</td>\n",
              "      <td>0.001392</td>\n",
              "      <td>0.177795</td>\n",
              "      <td>0.019919</td>\n",
              "      <td>0.028245</td>\n",
              "      <td>0.066984</td>\n",
              "      <td>0.095229</td>\n",
              "      <td>0.109740</td>\n",
              "      <td>0.017937</td>\n",
              "      <td>0.003561</td>\n",
              "      <td>0.111587</td>\n",
              "      <td>0.001606</td>\n",
              "      <td>0.027308</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2020</th>\n",
              "      <td>0.056979</td>\n",
              "      <td>0.091549</td>\n",
              "      <td>0.033705</td>\n",
              "      <td>0.053101</td>\n",
              "      <td>0</td>\n",
              "      <td>0.025264</td>\n",
              "      <td>0.073608</td>\n",
              "      <td>0.001067</td>\n",
              "      <td>0.165567</td>\n",
              "      <td>0.022092</td>\n",
              "      <td>0.025624</td>\n",
              "      <td>0.062998</td>\n",
              "      <td>0.111349</td>\n",
              "      <td>0.113050</td>\n",
              "      <td>0.019238</td>\n",
              "      <td>0.004743</td>\n",
              "      <td>0.107658</td>\n",
              "      <td>0.001867</td>\n",
              "      <td>0.030540</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2021</th>\n",
              "      <td>0.059928</td>\n",
              "      <td>0.094612</td>\n",
              "      <td>0.033794</td>\n",
              "      <td>0.050202</td>\n",
              "      <td>0</td>\n",
              "      <td>0.027117</td>\n",
              "      <td>0.076542</td>\n",
              "      <td>0.000837</td>\n",
              "      <td>0.177319</td>\n",
              "      <td>0.023231</td>\n",
              "      <td>0.026453</td>\n",
              "      <td>0.058440</td>\n",
              "      <td>0.095103</td>\n",
              "      <td>0.116620</td>\n",
              "      <td>0.018375</td>\n",
              "      <td>0.003009</td>\n",
              "      <td>0.108662</td>\n",
              "      <td>0.001448</td>\n",
              "      <td>0.028306</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2022</th>\n",
              "      <td>0.061361</td>\n",
              "      <td>0.095246</td>\n",
              "      <td>0.035829</td>\n",
              "      <td>0.047765</td>\n",
              "      <td>0</td>\n",
              "      <td>0.026121</td>\n",
              "      <td>0.074705</td>\n",
              "      <td>0.000819</td>\n",
              "      <td>0.178218</td>\n",
              "      <td>0.027006</td>\n",
              "      <td>0.024636</td>\n",
              "      <td>0.057157</td>\n",
              "      <td>0.094176</td>\n",
              "      <td>0.116181</td>\n",
              "      <td>0.018379</td>\n",
              "      <td>0.005657</td>\n",
              "      <td>0.106199</td>\n",
              "      <td>0.002315</td>\n",
              "      <td>0.028229</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2023</th>\n",
              "      <td>0.060552</td>\n",
              "      <td>0.094500</td>\n",
              "      <td>0.030900</td>\n",
              "      <td>0.046601</td>\n",
              "      <td>0</td>\n",
              "      <td>0.026286</td>\n",
              "      <td>0.076730</td>\n",
              "      <td>0.000653</td>\n",
              "      <td>0.182803</td>\n",
              "      <td>0.017661</td>\n",
              "      <td>0.024938</td>\n",
              "      <td>0.054866</td>\n",
              "      <td>0.100672</td>\n",
              "      <td>0.116431</td>\n",
              "      <td>0.017502</td>\n",
              "      <td>0.004823</td>\n",
              "      <td>0.109489</td>\n",
              "      <td>0.001884</td>\n",
              "      <td>0.032709</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c959abdc-ebb6-4851-b69a-f1f2fa973f2b')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-c959abdc-ebb6-4851-b69a-f1f2fa973f2b button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-c959abdc-ebb6-4851-b69a-f1f2fa973f2b');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-f2bc7da8-aeb9-400d-8813-9e239f2871c6\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-f2bc7da8-aeb9-400d-8813-9e239f2871c6')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-f2bc7da8-aeb9-400d-8813-9e239f2871c6 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "prop_summary_df",
              "summary": "{\n  \"name\": \"prop_summary_df\",\n  \"rows\": 6,\n  \"fields\": [\n    {\n      \"column\": \"ADJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0022087277739946533,\n        \"min\": 0.05566099715307687,\n        \"max\": 0.06136088148253306,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.05822981366459627,\n          0.05697913287923019,\n          0.05566099715307687\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADP\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0020203233979613974,\n        \"min\": 0.0905175560259873,\n        \"max\": 0.09524641542812838,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.09506853715999143,\n          0.09154863588856453,\n          0.0905175560259873\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"ADV\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.001618728808714689,\n        \"min\": 0.030899864342058984,\n        \"max\": 0.035829338342087735,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.03258192332405226,\n          0.03370454463545609,\n          0.03357909336447916\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"AUX\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0024065649415863206,\n        \"min\": 0.046601014922373514,\n        \"max\": 0.053101236169676,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.051269008352966376,\n          0.053101236169676,\n          0.048507190305861744\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"CONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"CCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0016857771966858688,\n        \"min\": 0.024089349587561135,\n        \"max\": 0.029047976011994003,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.029047976011994003\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"DET\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0016576276757108894,\n        \"min\": 0.07249946455343757,\n        \"max\": 0.07673047614262507,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.07249946455343757\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"INTJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0003027649399281384,\n        \"min\": 0.0006531678641410843,\n        \"max\": 0.00139216106232598,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.00139216106232598\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NOUN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.007174829430821198,\n        \"min\": 0.16556744873319637,\n        \"max\": 0.1828032624897419,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.1777950310559006\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"NUM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.003833774041326599,\n        \"min\": 0.016643550624133148,\n        \"max\": 0.027005776811942384,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.019918612122510174\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PART\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.00130793626279079,\n        \"min\": 0.024636082688128596,\n        \"max\": 0.028244806168344398,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.028244806168344398\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PRON\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.004747708002046039,\n        \"min\": 0.054866100587851074,\n        \"max\": 0.06698436496037695,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.06698436496037695\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PROPN\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.007859920708853715,\n        \"min\": 0.0941762309851157,\n        \"max\": 0.11134897466392764,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.09522917112872135\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PUNCT\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0033695811436667833,\n        \"min\": 0.10973977297065753,\n        \"max\": 0.11942477553106066,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.10973977297065753\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SCONJ\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.000777196485664588,\n        \"min\": 0.017008540769399227,\n        \"max\": 0.019238115832342237,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.017937459841507818\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SYM\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0015306670894522198,\n        \"min\": 0.0030093469119317614,\n        \"max\": 0.007299802905321557,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.00356071964017991\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"VERB\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0024314654279267507,\n        \"min\": 0.10619942559488059,\n        \"max\": 0.11270895685816483,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.11158706361105161\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"X\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0006787388477041993,\n        \"min\": 0.0014482066816801854,\n        \"max\": 0.003321410321921308,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.0016063396872992076\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SPACE\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0020273477809777283,\n        \"min\": 0.027307774684086527,\n        \"max\": 0.03270863688891122,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          0.027307774684086527\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Writing dataframes to excel spreadsheet"
      ],
      "metadata": {
        "id": "fKnspFqdzJ7q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output = '/content/drive/My Drive/fake_news_over_time/POS_frequency.xlsx'"
      ],
      "metadata": {
        "id": "tnd5Hb-LzJGq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Output is a single excel spreadsheet with the counts/proportion of tags of each year in separate tabs, and then two summary tabs at the end to compare across years."
      ],
      "metadata": {
        "id": "8h4FYUNECRIS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# create excel writer object to initialize new workbook\n",
        "writer = pd.ExcelWriter(output, engine=\"xlsxwriter\")\n",
        "\n",
        "# write dataframes to different worksheets\n",
        "counts_19_df.to_excel(writer, sheet_name=\"2019\", index=False)\n",
        "counts_20_df.to_excel(writer, sheet_name=\"2020\", index=False)\n",
        "counts_21_df.to_excel(writer, sheet_name=\"2021\", index=False)\n",
        "counts_22_df.to_excel(writer, sheet_name=\"2022\", index=False)\n",
        "counts_23_df.to_excel(writer, sheet_name=\"2023\", index=False)\n",
        "counts_24_df.to_excel(writer, sheet_name=\"2024\", index=False)\n",
        "\n",
        "counts_summary_df.to_excel(writer, sheet_name=\"counts_summary\")\n",
        "prop_summary_df.to_excel(writer, sheet_name=\"prop_summary\")\n",
        "\n",
        "# close the excel writer and output file\n",
        "writer.close()"
      ],
      "metadata": {
        "id": "rMHq9nfnzVo_"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}