change output folder to docs

.gitignore

@@ -2,7 +2,7 @@
 */.ipynb_checkpoints/*
 
 /.quarto/
-
+/_manuscript
 
 /agujournal2019.cls
 /trackchanges.sty

_freeze/notebooks/WHO-DON API/execute-results/html.json

@@ -1,8 +1,8 @@
 {
-  "hash": "bf72162cd5894ac9d6a0e14165d349ce",
+  "hash": "16b2eec0344296b8fc38127eb4a550a4",
   "result": {
     "engine": "knitr",
-    "markdown": "---\ntitle: Retrieving data from WHO-DON website\nauthor: Luong Nguyen Thanh\n---\n\n\n## Load necessary packages\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\npacman::p_load(\n  rio,\n  here,\n  httr,\n  jsonlite,\n  stringr,\n  tidyverse)\n```\n:::\n\n\n## Write a function to get data from WHO-DON website\n\nUsing method GET from `httr` package to get data from WHO-DON website. The function `get_who_news` will take a URL as input and return the data from the API. The function will return `NULL` if the request was unsuccessful.\n\nStatus code = 200 means the connection is successful. The function will parse the JSON content and return the data.\n\nNext, we will initialize the variables and loop to fetch all pages. The function will check if there is a next page link and keep fetching until there is no next page link.\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\n#| message: false\n#| warning: false\n#| eval: false\n\n# Function to get news data from a specific URL\nget_who_news <- function(url) {\n  # Make the GET request to the API\n  response <- GET(url)\n  \n  # Check the status of the response\n  if (status_code(response) == 200) {\n    # Parse the JSON content\n    content <- content(response, \"text\")\n    data <- fromJSON(content)\n    return(data)\n  } else {\n    # Return NULL if the request was unsuccessful\n    return(NULL)\n  }\n}\n\n# Initialize variables\nbase_url <- \"https://www.who.int/api/news/diseaseoutbreaknews\"\nall_news <- list()\nnext_url <- base_url\nkeep_fetching <- TRUE\n\n# Loop to fetch all pages\nwhile (keep_fetching) {\n  data <- get_who_news(next_url)\n  \n  if (!is.null(data) && \"value\" %in% names(data)) {\n    all_news <- c(all_news, data$value)\n    \n    # Check if there is a next page link\n    if (\"@odata.nextLink\" %in% names(data)) {\n      next_url <- data[[\"@odata.nextLink\"]]\n    } else {\n      keep_fetching <- FALSE\n    }\n  } else {\n    keep_fetching <- FALSE\n  }\n}\n```\n:::\n\n\n## Convert data from list to wide dataframe\n\nThe data is currently stored as a nested list. We will convert this nested list to a wide data frame for further analysis. We will define a function `convert_to_df` that takes the nested list as input and returns a data frame.\n\nSome cleaning steps are performed to remove HTML tags from the text data.\n\nFinally, we will export the data frame to a CSV file for further analysis.\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\n#| eval: false\n# Define a function to convert the nested list to a data frame\nconvert_to_df <- function(news_list) {\n  # Initialize an empty list to hold data frames\n  df_list <- list()\n  \n  # Determine the segment length\n  segment_length <- 22\n  \n  # Iterate through the list in steps of segment_length\n  for (i in seq(1, length(news_list), by = segment_length)) {\n    # Extract the current segment\n    segment <- news_list[i:(i + segment_length - 1)]\n    \n    # Convert the segment to a data frame and add it to the list\n    df_list[[length(df_list) + 1]] <- as.data.frame(segment)\n  }\n  \n  # Combine all data frames into one\n  combined_df <- do.call(rbind, df_list)\n  return(combined_df)\n}\n\n# Function to remove HTML tags\nremove_html_tags <- function(text) {\n  return(str_replace_all(text, \"<[^>]*>\", \"\"))\n}\n\nall_news_df <- convert_to_df(all_news)\n\n\nall_news_df %>% \n  mutate(across(where(is.character), remove_html_tags)) %>%\n  arrange(desc(PublicationDate)) %>%\n  rio::export(here(\"data\", \"who_dons.csv\"))\n```\n:::\n\n\n## Preview data\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\nall_news_df <- import(here(\"data\", \"who_dons.csv\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\nglimpse(all_news_df)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nRows: 3,102\nColumns: 22\n$ Id                     <chr> \"8c8a4612-7d82-4e67-adb8-e1bc9aa69c4b\", \"696077…\n$ LastModified           <dttm> 2024-05-30 14:31:48, 2024-05-16 14:53:26, 2024…\n$ PublicationDate        <dttm> 2024-05-30 10:31:02, 2024-05-08 16:40:02, 2024…\n$ DateCreated            <dttm> 2024-05-30 10:31:02, 2024-05-08 16:40:02, 2024…\n$ IncludeInSitemap       <lgl> TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE,…\n$ SystemSourceKey        <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…\n$ UrlName                <chr> \"2024-DON518\", \"2024-DON516\", \"2024-DON517\", \"2…\n$ ItemDefaultUrl         <chr> \"/2024-DON518\", \"/2024-DON516\", \"/2024-DON517\",…\n$ Response               <chr> \"The overall capacity for countries to respond …\n$ FurtherInformation     <chr> \"WHO Fact sheet: Dengue and severe dengue; http…\n$ Summary                <chr> \"As of 30 April 2024, over 7.6 million dengue c…\n$ PublicationDateAndTime <dttm> 2024-05-30 18:00:00, 2024-05-08 16:24:14, 2024…\n$ TitleSuffix            <chr> \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\",…\n$ UseOverrideTitle       <lgl> TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRU…\n$ Title                  <chr> \"Dengue - Global situation\", \"Middle East respi…\n$ Epidemiology           <chr> \"Dengue virus is transmitted to humans through …\n$ OverrideTitle          <chr> \"Dengue - Global situation\", \"Middle East respi…\n$ Advice                 <chr> \"Dengue is primarily an urban disease of the tr…\n$ Assessment             <chr> \"Dengue is a mosquito-borne viral disease cause…\n$ Overview               <chr> \"&nbsp;Global overviewCurrent situationAs of 30…\n$ DonId                  <chr> \"2024-DON518\", \"2024-DON516\", \"2024-DON517\", \"2…\n$ Provider               <chr> \"dynamicProvider372\", \"dynamicProvider372\", \"dy…\n```\n\n\n:::\n:::\n",
+    "markdown": "---\ntitle: Retrieving data from WHO-DON website\nauthor: Luong Nguyen Thanh\n---\n\n\n## Load necessary packages\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\npacman::p_load(\n  rio,\n  here,\n  httr,\n  jsonlite,\n  stringr,\n  tidyverse)\n```\n:::\n\n\n## Write a function to get data from WHO-DON website\n\nUsing method GET from `httr` package to get data from WHO-DON website. The function `get_who_news` will take a URL as input and return the data from the API. The function will return `NULL` if the request was unsuccessful.\n\nStatus code = 200 means the connection is successful. The function will parse the JSON content and return the data.\n\nNext, we will initialize the variables and loop to fetch all pages. The function will check if there is a next page link and keep fetching until there is no next page link.\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\n#| message: FALSE\n#| warning: FALSE\n#| eval: FALSE\n\n# Function to get news data from a specific URL\nget_who_news <- function(url) {\n  # Make the GET request to the API\n  response <- GET(url)\n  \n  # Check the status of the response\n  if (status_code(response) == 200) {\n    # Parse the JSON content\n    content <- content(response, \"text\")\n    data <- fromJSON(content)\n    return(data)\n  } else {\n    # Return NULL if the request was unsuccessful\n    return(NULL)\n  }\n}\n\n# Initialize variables\nbase_url <- \"https://www.who.int/api/news/diseaseoutbreaknews\"\nall_news <- list()\nnext_url <- base_url\nkeep_fetching <- TRUE\n\n# Loop to fetch all pages\nwhile (keep_fetching) {\n  data <- get_who_news(next_url)\n  \n  if (!is.null(data) && \"value\" %in% names(data)) {\n    all_news <- c(all_news, data$value)\n    \n    # Check if there is a next page link\n    if (\"@odata.nextLink\" %in% names(data)) {\n      next_url <- data[[\"@odata.nextLink\"]]\n    } else {\n      keep_fetching <- FALSE\n    }\n  } else {\n    keep_fetching <- FALSE\n  }\n}\n```\n:::\n\n\n## Convert data from list to wide dataframe\n\nThe data is currently stored as a nested list. We will convert this nested list to a wide data frame for further analysis. We will define a function `convert_to_df` that takes the nested list as input and returns a data frame.\n\nSome cleaning steps are performed to remove HTML tags from the text data.\n\nFinally, we will export the data frame to a CSV file for further analysis.\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\n#| eval: FALSE\n# Define a function to convert the nested list to a data frame\nconvert_to_df <- function(news_list) {\n  # Initialize an empty list to hold data frames\n  df_list <- list()\n  \n  # Determine the segment length\n  segment_length <- 22\n  \n  # Iterate through the list in steps of segment_length\n  for (i in seq(1, length(news_list), by = segment_length)) {\n    # Extract the current segment\n    segment <- news_list[i:(i + segment_length - 1)]\n    \n    # Convert the segment to a data frame and add it to the list\n    df_list[[length(df_list) + 1]] <- as.data.frame(segment)\n  }\n  \n  # Combine all data frames into one\n  combined_df <- do.call(rbind, df_list)\n  return(combined_df)\n}\n\n# Function to remove HTML tags\nremove_html_tags <- function(text) {\n  return(str_replace_all(text, \"<[^>]*>\", \"\"))\n}\n\nall_news_df <- convert_to_df(all_news)\n\n\nall_news_df %>% \n  mutate(across(where(is.character), remove_html_tags)) %>%\n  arrange(desc(PublicationDate)) %>%\n  rio::export(here(\"data\", \"who_dons.csv\"))\n```\n:::\n\n\n## Preview data\n\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\n#| echo: FALSE\nall_news_df <- import(here(\"data\", \"who_dons.csv\"))\n```\n:::\n\n::: {.cell}\n\n```{.r .cell-code .hidden}\nglimpse(all_news_df)\n```\n\n::: {.cell-output .cell-output-stdout}\n\n```\nRows: 3,102\nColumns: 22\n$ Id                     <chr> \"8c8a4612-7d82-4e67-adb8-e1bc9aa69c4b\", \"696077…\n$ LastModified           <dttm> 2024-05-30 14:31:48, 2024-05-16 14:53:26, 2024…\n$ PublicationDate        <dttm> 2024-05-30 10:31:02, 2024-05-08 16:40:02, 2024…\n$ DateCreated            <dttm> 2024-05-30 10:31:02, 2024-05-08 16:40:02, 2024…\n$ IncludeInSitemap       <lgl> TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE, TRUE,…\n$ SystemSourceKey        <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,…\n$ UrlName                <chr> \"2024-DON518\", \"2024-DON516\", \"2024-DON517\", \"2…\n$ ItemDefaultUrl         <chr> \"/2024-DON518\", \"/2024-DON516\", \"/2024-DON517\",…\n$ Response               <chr> \"The overall capacity for countries to respond …\n$ FurtherInformation     <chr> \"WHO Fact sheet: Dengue and severe dengue; http…\n$ Summary                <chr> \"As of 30 April 2024, over 7.6 million dengue c…\n$ PublicationDateAndTime <dttm> 2024-05-30 18:00:00, 2024-05-08 16:24:14, 2024…\n$ TitleSuffix            <chr> \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\", \"\",…\n$ UseOverrideTitle       <lgl> TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRU…\n$ Title                  <chr> \"Dengue - Global situation\", \"Middle East respi…\n$ Epidemiology           <chr> \"Dengue virus is transmitted to humans through …\n$ OverrideTitle          <chr> \"Dengue - Global situation\", \"Middle East respi…\n$ Advice                 <chr> \"Dengue is primarily an urban disease of the tr…\n$ Assessment             <chr> \"Dengue is a mosquito-borne viral disease cause…\n$ Overview               <chr> \"&nbsp;Global overviewCurrent situationAs of 30…\n$ DonId                  <chr> \"2024-DON518\", \"2024-DON516\", \"2024-DON517\", \"2…\n$ Provider               <chr> \"dynamicProvider372\", \"dynamicProvider372\", \"dy…\n```\n\n\n:::\n:::\n",
     "supporting": [],
     "filters": [
       "rmarkdown/pagebreak.lua"