02_Data_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<img style=\"float: center;\" src=\"images/CI_horizontal.png\" width=\"600\">\n",
    "</center>\n",
    "<center>\n",
    "    <span style=\"font-size: 1.5em;\">\n",
    "        <a href='https://www.coleridgeinitiative.org'>Website</a>\n",
    "    </span>\n",
    "</center>\n",
    "<center> Julia Lane, Brian Kim, Benjamin Feder, Ekaterina Levitskaya. </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Visualization in R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook covers how to use different visualization methods in R to further explore and analyze data, with a particular focus on how to clearly and accurately communicate results using labels and various plotting adjustments.\n",
    "\n",
    "The following methods are displayed and discussed:\n",
    "- **Histogram** \n",
    "(visualizing distributions, continuous variables)\n",
    "- **Small multiples**\n",
    "(using a series of mini-graphs to compare information by different groups)\n",
    "- **Bar plot**\n",
    "(visualizing relationships between numerical and categorical variables)\n",
    "- **Heatmap** \n",
    "(adding highlights to the data with color-coding)\n",
    "\n",
    "For all visualizations, the R package`ggplot2` is used (`ggplot2` is included in the `tidyverse` suite of packages). The syntax of `ggplot2` in most cases is consistent:\n",
    "\n",
    "- start with `ggplot()` <br>\n",
    "- then, supply a dataset and aesthetic mapping with `x` pertaining to the variable on the x-axis, and so on, for example: `ggplot(dataset, aes(x = ..., y = ...)` <br>\n",
    "- from there, provide a geometric object represented by `geom_` to convey the desired type of visualization <br>\n",
    "- finally, add additional layers if necessary using `+` <br>\n",
    " \n",
    "For example, the code below can be used\n",
    "\n",
    "    ggplot(data, aes(x = ... , y = ...) + \n",
    "        geom_histogram()\n",
    "    \n",
    "to create a histogram (geometric object), or with the addition of an extra layer\n",
    "\n",
    "    ggplot(data, aes(x = ... , y = ...) + \n",
    "        geom_histogram() + \n",
    "        labs(title = 'My plot title')\n",
    "        \n",
    "to create a histogram with a title for the graph, and so on. </center>\n",
    "\n",
    "`ggplot()` can work together with other tidy functions, as `%>%` can be used to set the desired data set as the `data` argument in the `ggplot()` call. (Wickham 2016)\n",
    "\n",
    "> Note: The `aes()` call can contain additional arguments outside of `x` and `y` to potentially match the `fill`, `color`, `linetype`, and additional specifications of specific variables in a dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Motivation\n",
    "\n",
    "In answering a series of specific questions about the data, various visualizations are introduced to provide a clearer view of the data than just using summary statistics. These powerful graphics may better convey conclusions from an analysis compared to the previously introduced summary statistics in the Data Exploration notebook.\n",
    "\n",
    "The questions covered in this notebook are:\n",
    "- How long does it take students to get their PhDs? That is, what is the distribution of time to degree? How does this differ by primary source of support?\n",
    "- What are the primary sources of support for students, and how do they differ by field of study?\n",
    "- What are the federal funding histories of graduate students in the three years leading up to their dissertation? How do the funding histories differ by the number of semesters of federal funding?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### R Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, the code for this notebook starts by importing necessary R libraries, setting the viewing options, and connecting to the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#database interaction imports\n",
    "library(DBI)\n",
    "library(RPostgreSQL)\n",
    "\n",
    "# for data manipulation/visualization\n",
    "library(tidyverse)\n",
    "library(lubridate)\n",
    "\n",
    "# for calculating percentages\n",
    "library(scales)\n",
    "\n",
    "# to better view images\n",
    "# For easier viewing of graphs\n",
    "# Adjust repr.plot.width and repr.plot.height to change the size of graphs\n",
    "theme_set(theme_gray(base_size = 24))\n",
    "options(repr.plot.width = 20, repr.plot.height = 12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an RPostgreSQL driver\n",
    "drv <- dbDriver(\"PostgreSQL\")\n",
    "\n",
    "# connect to the database\n",
    "con <- dbConnect(drv,dbname = \"postgresql://stuffed.adrf.info/appliedda\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that `appliedda` database has been properly connected to, the 2015 SED cohort, which was also used in the Data Exploration notebook, can be loaded into R. This cohort is also the main data subject here. As a reminder, this cohort consists of doctorate students included in the Survey of Earned Doctorates that completed their degree in 2015. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read cohort of all SED 2015 graduates into R\n",
    "qry <- \"\n",
    "select *\n",
    "from ncses_2019.nsf_sed\n",
    "WHERE phdfy = '2015'\n",
    "\"\n",
    "\n",
    "sed_ncses_2015 <- dbGetQuery(con, qry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned in the Data Exploration notebook, running `head()` on a data frame will output the first six rows of the dataset. The code may be uncommented by deleting the pound sign, which allows for the viewing of the output from `head()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# can see first six rows of sed_ncses_2015\n",
    "# head(sed_ncses_2015)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time to Doctoral Degree\n",
    "\n",
    "The SED data contains a variety of information about each graduate, including their time to finish their doctoral degree, which is the `ttddoc` variable in the `sed_ncses_2015` data frame.\n",
    "\n",
    "This question can be answered in different ways depending on the desired output, such as by using a measure of centrality (mean, median, mode), providing specific percentile cutoffs, etc. However, to view the entire distribution of time to degree, it may make sense to plot a histogram, which shows counts of entries for the entire range of the numerical variable. \n",
    "\n",
    "> In some cases, time to degree can be a categorical variable, such as when each value corresponds to a specific range, i.e. <4, 5-6, 7-8, etc., but that is not the case here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a histogram in the `ggplot2` library, the steps listed [above](#Introduction) are followed. In this example, the R object `sed_ncses_2015` functions as the dataset, the `ttddoc` variable corresponds to the variable on the x-axis, and the geometric object to be plotted in R is `geom_histogram()`. \n",
    "\n",
    "> The counts do not need to be designated on the y-axis since the `geom_histogram()` automatically computes the counts on the y-axis for the variable supplied on the x-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# intial histogram\n",
    "sed_ncses_2015 %>%\n",
    "    ggplot(aes(x=ttddoc)) +\n",
    "    geom_histogram()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a reminder, the code displayed above can also be written as:\n",
    "\n",
    "    ggplot(sed_ncses_2015, aes(x = ttddoc)) +\n",
    "        geom_histogram()`\n",
    "    \n",
    "The basic code can also be saved to an object in R, which can then be supplied additional layers to improve the visuals of the original plot. In this case, the code above is saved to the R object `ttd_graph`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save code above to ttd_graph\n",
    "ttd_graph <- sed_ncses_2015 %>%\n",
    "    ggplot(aes(x=ttddoc)) +\n",
    "    geom_histogram() \n",
    "\n",
    "# can print ttd_graph to see plot\n",
    "ttd_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ggplot2` object contains a medley of potential layers. A `labs()` layer can be added to the original visualization to change the title, axes' names, as well as the caption of the plot. Again, this additional layer can be added by using `+`.\n",
    "\n",
    "The title of the visualization should convey the major takeaway(s) of the plot and should answer the original question \"About how long do graduate students take to finish their dissertations?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add titles and caption\n",
    "ttd_graph +\n",
    "    labs(\n",
    "        title = \"Most graduate students finished their dissertations in around REDACTED years, but some take much longer\",\n",
    "        y = 'Number of dissertators',\n",
    "        x='Time to completion',\n",
    "        caption = 'Source: NCSES SED data'\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example can be extended a bit further. For example, a natural question to ask might be whether this distribution changes by the primary source of support.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Differ by primary source of support"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The distinct values that `srceprim` can take on in the cohort (`sed_ncses_2015`) can be found using `distinct()`. The resulting table is a data frame with each row representing a different value of `srceprim` found in `sed_ncses_2015`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find distinct source of support names\n",
    "sed_ncses_2015 %>%\n",
    "    distinct(srceprim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that we added the descriptions in the Data Exploration notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an empty dataframe and fill it with the primary source of support letter and description\n",
    "primary_source <- data.frame(\"srceprim\" = c(\"A\",\"B\",\"C\",\"D\",\"E\",\"F\",\"G\",\"H\",\"I\",\"J\",\"K\",\"L\",\"M\",\"N\"),\n",
    "                            \"description\" = c('Fellowship, scholarship', 'Dissertation grant', 'Teaching assistantship', 'Research assistantship',\n",
    "                                'Other assistantship','Traineeship','Internship, clinical residency', 'Loans',\n",
    "                                'Personal savings', 'Personal earnings',\n",
    "                                'Spouse/partner/family', 'Employer reimbursement',\n",
    "                                'Foreign (non-U.S.) support', 'Other')\n",
    "                            )\n",
    "primary_source"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The base graph that we will use can be re-created as above, but this time also including the merged primary source of support description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ttd_graph <- sed_ncses_2015 %>%\n",
    "    merge(primary_source,by = 'srceprim') %>%\n",
    "    ggplot(aes(x=ttddoc)) +\n",
    "    geom_histogram() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The addition of a `facet_wrap()` layer to the existing `ttd_graph` object creates a separate panel for each distinct value of the variable(s) provided. In this case, the variable would be `description`, and the expectation is to see REDACTED separate time to degree distributions, each panel for all individuals in `sed_ncses_2015` within the different `srceprim` values. \n",
    "\n",
    "The `~` is used to designate the variable(s) upon which to create the separate panels.\n",
    "\n",
    "> There also exists a layer `facet_grid()`, which is more useful for displaying differences across different discrete variables, as opposed to within all levels of one variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add on facet_wrap by primary source of support\n",
    "ttd_graph +\n",
    "    facet_wrap(~description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limiting the number of categories displayed can be done by using the `filter` function on the original data frame, so that only a select few categories are displayed for easier comparison. The categories to include are stored inside `support_categories` and the `%in%` is used to filter the data frame so that only those categories are kept. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "support_categories <- c('Research assistantship','Teaching assistantship', 'Loans', 'Personal earnings', 'Personal savings', 'Employer reimbursement')\n",
    "\n",
    "ttd_graph <- sed_ncses_2015 %>%\n",
    "    merge(primary_source,by = 'srceprim') %>%\n",
    "    filter(description %in% support_categories) %>%\n",
    "    ggplot(aes(x=ttddoc)) +\n",
    "    geom_histogram() \n",
    "\n",
    "ttd_graph + \n",
    "    facet_wrap(~description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The titles and other general aspects of the visualization can still be modified by adding on the `labs()` layer to the plot call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add in titles\n",
    "ttd_graph + \n",
    "    facet_wrap(~description) + \n",
    "    labs(\n",
    "        title = \"Lower variance in time to degree for students supported by REDACTED\",\n",
    "        y = 'Number of dissertators',\n",
    "        x = 'Time to degree',\n",
    "        caption = 'Source: NCSES SED data'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'>Checkpoint: Graphing Distributions</font>\n",
    "\n",
    "Examine the distribution of `ttddoc` by a different categorical variable, such as sex (which is represented by the `sex` variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Graphing the Distribution of a Categorical Variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Source of Funding\n",
    "\n",
    "The last section discussed how a histogram may be optimal for visualizing the distribution of a numerical variable. This example pivots to graphing a categorical variable (primary source of funding). Since the desired plot calls for one value (the count), a histogram will not be effective. Instead, this visualization can be created effectively using a bar chart, which allows for a specific numerical variable on one axis, and the categorical one on the other.\n",
    "\n",
    "Take note of the motivation behind each visualization - what is the reasoning for it? Which findings should be easily conveyed? For this example, an effective visualization would clearly display the differences in the primary source of funding in the original 2015 SED cohort.\n",
    "\n",
    "The first step is to look at a table of the values. This can be done using the `group_by` and `summarize` functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sed_ncses_2015 %>% group_by(srceprim) %>% summarize(n = n())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These codes for primary source of support aren't very informative. The descriptions of each of these can be added using the same merging method demonstrated above and in the Data Exploration notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "support_count <- sed_ncses_2015 %>% group_by(srceprim) %>% summarize(n = n()) %>% merge(primary_source, by = 'srceprim')\n",
    "support_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The`arrange` function can then be used to order the values in decreasing order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "support_count %>% arrange(desc(n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the information needed to create the initial plot is available in `support_count`. In other words, this means that `support_count` can function as the dataset to pipe into the `ggplot()` call. Instead of adding on the `geom_histogram()` geometric object, a barplot can be created with the `geom_col()` layer. As before, variables corresponding to the x-axis (and in this case also the y-axis) can be assigned inside the `aes()` call inside `ggplot()`. Additionally, when creating barplots, if the desired numerical output is a value in the dataframe, as opposed to the count within each grouping, the geometric object used should be `geom_col()` instead of `geom_bar()`, which will be used later.\n",
    "\n",
    "In this example, `n` goes on the x-axis and `description` on the y-axis. This creates a horizontal bar chart. To create a vertical bar chart, simply switch the x and y arguments, so that `n` goes on the y-axis and `description` on the x-axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# finding counts by primary source of support\n",
    "support_count %>%\n",
    "    ggplot(aes(x=n, y =description)) + \n",
    "    geom_col()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To help convey the differences a bit more, the bars on the y-axis can be reordered from highest to lowest by adding `reorder()` to the `y` argument inside the `aes()` call, with an additional argument describing the variable upon which to reorder, which here would be `n`. \n",
    "\n",
    "Since it is now just a matter of adding separate layers to make the visualization easier to read, this base visualization is saved to the R object `support_graph`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reorder y-axis\n",
    "support_graph <- support_count %>%\n",
    "    ggplot(aes(x=n, y =reorder(description,n))) + \n",
    "    geom_col()\n",
    "support_graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, proper descriptions of the legend, x-axis, y-axis, title, and caption for the figure can be added. Additionally, the legend with repetitive information can be removed to allow for more width to be devoted to the differences between the primary source of support categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# graph with titles \n",
    "support_graph +\n",
    "    guides(fill=guide_legend(title=\"Primary Source of Support\")) +\n",
    "    theme(legend.position=\"none\") + \n",
    "    labs(\n",
    "        x = 'Count',\n",
    "        y = 'Primary Source of Support',\n",
    "        title = 'REDACTED are the most common form of support',\n",
    "        caption = 'SED NCSES Data'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'> Checkpoint: Graphs of Categorical Variables </font>\n",
    "    \n",
    "Try creating a bar chart of counts of another categorical variable, such as sex or race/ethnicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Primary Funding Source and PhD Fields: Graphing Relationships Using Heatmaps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `srceprim` variable contains self-reported primary source of support from the SED data. Suppose there was an interest in looking at how this differed according to the PhD Field. This question could potentially be answered using a large contingency table, but it might be very hard to interpret because there are so many numbers. This is where something like a heatmap can come in and be useful, by drawing the eye to key differences.\n",
    "\n",
    "To get a better sense of the variables at hand, the code below draws a few rows from both the `srceprim` and `phdfield_name` variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see srceprim and phdfield_name\n",
    "sed_ncses_2015 %>%\n",
    "    select(srceprim, phdfield_name) %>%\n",
    "    head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As per the data dictionary, unknown funding sources for `srceprim` are coded as ` `. These entries will be coded as `NA`. Afterwards, the amount of entries within each `srceprim`/`phdfield_name` combination will be calculated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate known srceprim/phdfield_name combinations\n",
    "funding_phdfield <- sed_ncses_2015 %>%\n",
    "    filter(srceprim != '') %>%\n",
    "    count(srceprim, phdfield_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of visualizing differences in counts between the different fields, it may be more effective to view the row proportions to see how fields differ by primary funding sources. In other words, these are the proportions of primary source of funding within each field. This will allow for a distinct differentiation between how the funding sources are distributed within each field, normalizing by the number of students in each field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find proportion of primary funding sources within each field\n",
    "funding_phdfield <- sed_ncses_2015 %>%\n",
    "    filter(srceprim != '') %>% \n",
    "    group_by(phdfield_name,srceprim)%>%\n",
    "    summarize(n = n()) %>%\n",
    "    mutate(proportion = n/sum(n))\n",
    "\n",
    "# see first six rows\n",
    "#head(funding_phdfield)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, the `primary_source` data frame is then merged with the `funding_phdfield` to add in the descriptions (see previous section for how `primary_source` was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "funding_phdfield <- merge(funding_phdfield, primary_source, by = 'srceprim')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head(funding_phdfield)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `funding_phdfield` data frame contains all of the information required to plot the heatmap now. Instead of using `geom_histogram()` or `geom_bar()` as the geometric object, heatmaps can be created using `geom_tile()`. When creating a heatmap, it makes sense to use three aesthetics in mapping - an x-axis, y-axis and fill variables. In this case, `srceprim` and `phdfield_name` will be assigned to the `x` and `y` axes, respectively, while the `fill` will vary by `proportion` in `funding_phdfield`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot base heatmap\n",
    "funding_phdfield %>%\n",
    "    ggplot(aes(x=description, y=phdfield_name, fill=proportion)) + \n",
    "    geom_tile() +\n",
    "    theme(axis.text.x = element_text(angle = 90))\n",
    "\n",
    "viz <- funding_phdfield %>%\n",
    "    ggplot(aes(x=description, y=phdfield_name, fill=proportion)) + \n",
    "    geom_tile() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, labels can be added to a heatmap by adding the layer `labs()`. Additionally, the color scale of the heatmap can be changed by adding a `scale_fill_gradient...()` layer - `scale_fill_gradient2()` creates a diverging colour gradient, according to its documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change fill scale and add labels\n",
    "viz +\n",
    "    scale_fill_gradient2() + \n",
    "    labs(\n",
    "        x = 'Primary Funding Source (SED)',\n",
    "        y = 'PhD Field', \n",
    "        title = 'REDACTED PhDs Primarily Supported by REDACTED,\\n REDACTED PhDs by REDACTED',\n",
    "        caption = \"Source: SED NCSES Data\"\n",
    "    )+\n",
    "    theme(axis.text.x = element_text(angle = 90))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Federal Funding Histories: Working with UMETRICS \n",
    "\n",
    "One of the main advantages of the UMETRICS data is that it provides information on graduate student funding histories at the transaction level, much more granular than that in the SED, not to mention likely more accurate. In this section, the funding histories of the cohort will be explored to see how they might differ in terms of how many semesters they were supported by federal funding. \n",
    "\n",
    "Note that the UMETRICS data has already been aggregated to the person-semester level in the `iris_semester` table. Only some institutions have been covered in the UMETRICS data from 2012-2015, and since the example is about funding patterns, the original cohort from the SED data will be subset to contain individuals who received their doctorate at one of these schools.\n",
    "\n",
    "Ultimately, since each potential number of semesters on federal funding (0-12) functions as a separate group, and the distribution of the number of individuals by the number of semesters they received federal funding is desired, a barplot may best display this data (a numerical variable within groups)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# filter for institutions with 2012-2015 UMETRICS data coverage\n",
    "qry <- \"\n",
    "select *\n",
    "from ncses_2019.nsf_sed\n",
    "WHERE phdfy = '2015' and \n",
    "phdinst in (REDACTED)\n",
    "\"\n",
    "sed_upd_cohort <- dbGetQuery(con, qry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see amount of people in cohort\n",
    "sed_upd_cohort %>% \n",
    "    summarize(n_distinct(drf_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the Data Exploration notebook, there is not a direct link between the SED and UMETRICS data. Luckily, the `sed_umetrics_xwalk` table in `ncses_2019` provides a link between the two tables. The only two variables from `iris_semester` used in this example are the `semester` and `team_size` ones, as `semester` designates the semester when the individual received funding, and `team_size` indicates whether the individual received federal funding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# join sed to umetrics using umetrics_xwalk\n",
    "qry <- \"\n",
    "select c.*, d.semester, d.team_size from (\n",
    "select a.*, b.emp_number \n",
    "from ncses_2019.nsf_sed a\n",
    "inner join ncses_2019.sed_umetrics_xwalk b\n",
    "on a.drf_id = b.drf_id\n",
    "where a.phdfy = '2015' and a.phdinst in (REDACTED) ) c\n",
    "inner join ncses_2019.iris_semester d\n",
    "on c.emp_number = d.emp_number\n",
    "\"\n",
    "cohort_joined <- dbGetQuery(con, qry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see cohort_joined\n",
    "#head(cohort_joined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some individuals may have more than one row now, as one row of `cohort_joined` corresponds to a unique individual and semester of funding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see number of rows relative to individuals\n",
    "cohort_joined %>%\n",
    "    summarize(\n",
    "        n_rows=n(), \n",
    "        n_people = n_distinct(drf_id)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An individual only received federal funding if the `team_size` in the specific semester is at least 1. Within the UMETRICS data, the only time `team_size` is ever less than 1 is if it were coded as `NA`. Additionally, the `team_size` values can change for a single individual by each semester. \n",
    "\n",
    "Therefore, the number of quarters each individual received federal funding can be calculated in a three step process: \n",
    "- Calculating the number of semesters for each `drf_id` and `team_size` combination (again, there may be multiple combinations for a single `drf_id`)\n",
    "- For the rows where `team_size` is `NA`, replace the number of semesters of federal funding with 0\n",
    "- Total up the newly-updated number of semesters of federal funding by each `drf_id` and `team_size` based on each `drf_id`\n",
    "\n",
    "The following process is implemented below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find number of semesters of federal funding per person\n",
    "fed_sems <- cohort_joined %>%\n",
    "    group_by(drf_id, team_size) %>%\n",
    "    summarize(n_sems_fed = n_distinct(semester)) %>%\n",
    "    ungroup() %>%\n",
    "    mutate(\n",
    "        n_sems_fed = ifelse(is.na(team_size), 0, n_sems_fed)\n",
    "    ) %>%\n",
    "    group_by(drf_id) %>%\n",
    "    summarize(\n",
    "        fed_funds = sum(n_sems_fed)\n",
    "    ) %>%\n",
    "    ungroup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the information required to generate the desired barplot exists in `fed_sems`, `fed_sems` can be used as the data frame within the `ggplot()` to create a barplot. \n",
    "\n",
    "However, since this visualization will not display any specific values in the data frame, as shown in the \"Primary source of support\" [example](05_Data_Visualization.ipynb/#Primary-source-of-support) but rather the counts, or proportions of individuals by the number of semesters in which they received federal funding, the geometric object used will be `geom_bar()` to create this bar chart, as opposed to `geom_col()` above. In fact, `fed_sems` does not need to be aggregated to find the counts, or proportions, by each potential number of semesters of federal funding, due to the existence of special variables in `ggplot2`, which will automatically calculate the proportion, or count, or other transformations, within each of the groupings provided on the other axis.\n",
    "\n",
    "This example will demonstrate how to use `..prop..` as a special variable for the y-axis. \n",
    "\n",
    "> All special variables in `ggplot2` are designated with `..` before and after the variable, e.g. `..count..` and `..prop..`. This bar plot can also be generated by directly computing the proportions within each group of semester in `fed_sems` instead of using `..prop..` as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initial plot of proportions of individuals that received a certain number of semesters of federal funding\n",
    "fed_sems %>%\n",
    "    ggplot(aes(x=fed_funds, y=..prop..)) +\n",
    "    geom_bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The current y-axis's `..prop..` can be transformed into a percentage by multiplying `..prop..` by 100. This will serve as the base `viz` object upon which separate layers can be added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform ..prop.. variable\n",
    "prop_fed_funds <- fed_sems %>%\n",
    "    ggplot(aes(x=fed_funds, y=..prop..*100)) +\n",
    "    geom_bar()\n",
    "prop_fed_funds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, to aid the audience in deciphering the specific percentages within each bar, the percentages can be added using the `geom_text()` layer. Similar to the `ggplot()` function, which works with `aes()`, `geom_text()` requires a specific aesthetic to display the proper `label`, or percentage, which in this case will be the percentage, found using the `percent()` function from the `scales` library, of `..prop..`. Additionally, since the visualization uses a special variable, the `stat` argument must be set to `\"count\"` to designate that `..prop..` is based off of counts within groups, or `fed_funds`. Finally, the `vjust` argument is added to easily view the percentages above each bar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add in percentages above the bars\n",
    "prop_fed_funds +\n",
    "    geom_text(aes(label=percent(..prop.., .01)), stat=\"count\", vjust = -.5) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before adding in the proper labels, to have each tick on the x-axis correspond to a specific bar, since there are 13 potential bars, ranging from 0 to 12, the `scale_x_continuous()` layer can be added.\n",
    "\n",
    "> `seq(0, 12, by = 1)` creates a vector of numbers from 0 to 12, with the break in between numbers being 1, i.e. 0, 1, 2, ..., 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change x-axis ticks\n",
    "prop_fed_funds +\n",
    "    geom_text(aes(label=percent(..prop.., .01), y=..prop..*100), stat=\"count\", vjust = -.5) +\n",
    "    scale_x_continuous(breaks = seq(0,12, by=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the labels for the axes, title, and caption can be added as per usual with `labs()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# add in labels\n",
    "prop_fed_funds +\n",
    "    geom_text(aes(label=percent(..prop.., .01), y=..prop..*100), stat=\"count\", vjust = -.5) +\n",
    "    scale_x_continuous(breaks = seq(0,12, by=1)) + \n",
    "    labs(\n",
    "        x = \"Number of Semesters Receiving Federal Funding\",\n",
    "        y = \"Percent\",\n",
    "        title = \"Most PHD Candidates did not Receive more than REDACTED semesters of Federal Funding\",\n",
    "        caption = \"Source: SED NCSES and UMETRICS data\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is such an incredible variety of potential layers to add onto any base visualization in `ggplot2`, and while this notebook covers just a few of them, it is recommended to look at the sheer breadth of layering options, which is just one of the reasons as to why `ggplot2` can be such a valuable library in R when visualizing information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color='red'>Checkpoint</font>\n",
    "\n",
    "Try using small multiples to look at the number of semesters receiving federal funding by Phd field, gender, or race/ethnicity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Additional Visualizations\n",
    "\n",
    "Additional visualization notebooks for more advanced techniques are provided in the Supplemental_Visualizations notebook. That notebook builds on the concepts covered in this notebook, and expand on how to manipulate the tools available in `ggplot2` to create various visualizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "Wickham H (2016). _ggplot2: Elegant Graphics for Data Analysis._ Springer-Verlag New York. ISBN 978-3-319-24277-4, https://ggplot2.tidyverse.org"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "adrf_r"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.0.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}