L4_testing_framework.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "893d5a85-8e19-40f7-ab20-5660e2820614",
   "metadata": {},
   "source": [
    "# Lesson 4: Comprehensive Testing Framework"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7afdae8b-82b8-4005-93c0-be04f536c97f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3850d00b",
   "metadata": {},
   "source": [
    "## Import the API keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36d25a8c-70bc-4933-8b09-d70d6de37ea4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_circle_api_key\n",
    "cci_api_key = get_circle_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea1d3be-5db8-410d-b1fc-9004d4ad0ca9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_gh_api_key\n",
    "gh_api_key = get_gh_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6036ffc6-9e89-4058-87b3-642f18941c94",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_openai_api_key\n",
    "openai_api_key = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47797497",
   "metadata": {},
   "source": [
    "## Set up our github branch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3279b7b2-cfe2-4d5c-8ba0-f506f66372d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_repo_name\n",
    "course_repo = get_repo_name()\n",
    "course_repo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67480d99-dd05-450a-829e-8f9b90333b8e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import get_branch\n",
    "course_branch = get_branch()\n",
    "course_branch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78df18ff",
   "metadata": {},
   "source": [
    "Aditionally, the file quiz_bank.txt should be uploaded to this workspace (already done!). This file has the information about the content we want for the quizzes generated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ee701c0-d3ac-45e8-81d7-040dd3cceb3d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import read_file_into_string\n",
    "quiz_bank = read_file_into_string(\"quiz_bank.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5dcb9ea-9905-4e3f-9e60-e47d4e340e95",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(quiz_bank)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc155227",
   "metadata": {},
   "source": [
    "## Rebuild the quiz generator app"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "179b2db4-ccec-4483-ab59-3a279250bbb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "delimiter = \"####\"\n",
    "system_message = f\"\"\"\n",
    "Follow these steps to generate a customized quiz for the user.\n",
    "The question will be delimited with four hashtags i.e {delimiter}\n",
    "\n",
    "Step 1:{delimiter} First identify the category user is asking about from the following list:\n",
    "* Geography\n",
    "* Science\n",
    "* Art\n",
    "\n",
    "Step 2:{delimiter} Determine the subjects to generate questions about. The list of topics are below:\n",
    "\n",
    "{quiz_bank}\n",
    "\n",
    "Pick up to two subjects that fit the user's category.\n",
    "\n",
    "Step 3:{delimiter} Generate a quiz for the user. Based on the selected subjects generate 3 questions for the user using the facts about the subject.\n",
    "* Include any facts that might be interesting\n",
    "\n",
    "Use the following format:\n",
    "Question 1:{delimiter} <question 1>\n",
    "\n",
    "Question 2:{delimiter} <question 2>\n",
    "\n",
    "Question 3:{delimiter} <question 3>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f67587d5",
   "metadata": {},
   "source": [
    "## Create Chat Prompt\n",
    "Create function assistant_chain() to build the chat_prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aede34b-5d44-42b2-975b-d647379221e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts                import ChatPromptTemplate\n",
    "from langchain.chat_models            import ChatOpenAI\n",
    "from langchain.schema.output_parser   import StrOutputParser\n",
    "\n",
    "def assistant_chain():\n",
    "  human_template  = \"{question}\"\n",
    "\n",
    "  chat_prompt = ChatPromptTemplate.from_messages([\n",
    "      (\"system\", system_message),\n",
    "      (\"human\", human_template),\n",
    "  ])\n",
    "  return chat_prompt | \\\n",
    "         ChatOpenAI(model=\"gpt-3.5-turbo\", \n",
    "                    temperature=0) | \\\n",
    "         StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd2aceeb",
   "metadata": {},
   "source": [
    "# Hallucination detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c022d3b3-6f7d-4094-abbf-9bdff4ee7ae3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts                import ChatPromptTemplate\n",
    "from langchain.chat_models            import ChatOpenAI\n",
    "from langchain.schema.output_parser   import StrOutputParser\n",
    "\n",
    "def create_eval_chain(context, agent_response):\n",
    "  eval_system_prompt = \"\"\"You are an assistant that evaluates \\\n",
    "  how well the quiz assistant\n",
    "    creates quizzes for a user by looking at the set of \\\n",
    "    facts available to the assistant.\n",
    "    Your primary concern is making sure that ONLY facts \\\n",
    "    available are used. Quizzes that contain facts outside\n",
    "    the question bank are BAD quizzes and harmful to the student.\"\"\"\n",
    "  \n",
    "  eval_user_message = \"\"\"You are evaluating a generated quiz \\\n",
    "  based on the context that the assistant uses to create the quiz.\n",
    "  Here is the data:\n",
    "    [BEGIN DATA]\n",
    "    ************\n",
    "    [Question Bank]: {context}\n",
    "    ************\n",
    "    [Quiz]: {agent_response}\n",
    "    ************\n",
    "    [END DATA]\n",
    "\n",
    "Compare the content of the submission with the question bank \\\n",
    "using the following steps\n",
    "\n",
    "1. Review the question bank carefully. \\\n",
    "  These are the only facts the quiz can reference\n",
    "2. Compare the quiz to the question bank.\n",
    "3. Ignore differences in grammar or punctuation\n",
    "4. If a fact is in the quiz, but not in the question bank \\\n",
    "   the quiz is bad.\n",
    "\n",
    "Remember, the quizzes need to only include facts the assistant \\\n",
    "  is aware of. It is dangerous to allow made up facts.\n",
    "\n",
    "Output Y if the quiz only contains facts from the question bank, \\\n",
    "output N if it contains facts that are not in the question bank.\n",
    "\"\"\"\n",
    "  eval_prompt = ChatPromptTemplate.from_messages([\n",
    "      (\"system\", eval_system_prompt),\n",
    "      (\"human\", eval_user_message),\n",
    "  ])\n",
    "\n",
    "  return eval_prompt | ChatOpenAI(\n",
    "      model=\"gpt-3.5-turbo\", \n",
    "      temperature=0) | \\\n",
    "    StrOutputParser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4e5f93d",
   "metadata": {},
   "source": [
    "Create function test_model_graded_eval_hallucination(quiz_bank) to test about hallucinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85637c83-5daa-4db7-93ca-f16a971556e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_model_graded_eval_hallucination(quiz_bank):\n",
    "  assistant = assistant_chain()\n",
    "  quiz_request = \"Write me a quiz about books.\"\n",
    "  result = assistant.invoke({\"question\": quiz_request})\n",
    "  print(result)\n",
    "  eval_agent = create_eval_chain(quiz_bank, result)\n",
    "  eval_response = eval_agent.invoke({\"context\": quiz_bank, \"agent_response\": result})\n",
    "  print(eval_response)\n",
    "  # Our test asks about a subject not in the context, so the agent should answer N\n",
    "  assert eval_response == \"N\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc9c2322",
   "metadata": {},
   "source": [
    "Try test_model_graded_eval_hallucination function with the quiz_bank (mentioned before)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c68ba32-9754-4c2a-89d3-2fe6a98b1162",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_model_graded_eval_hallucination(quiz_bank)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a284a91",
   "metadata": {},
   "source": [
    "## Reviewing evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7ce5409",
   "metadata": {},
   "source": [
    "Improve the eval_system_prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "460fd7ff-135e-43e0-b383-2c7df2dcd89e",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_system_prompt = \"\"\"You are an assistant that evaluates \\\n",
    "how well the quiz assistant\n",
    "    creates quizzes for a user by looking at the set of \\\n",
    "    facts available to the assistant.\n",
    "    Your primary concern is making sure that ONLY facts \\\n",
    "    available are used.\n",
    "    Helpful quizzes only contain facts in the test set.\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a5d7fb4",
   "metadata": {},
   "source": [
    "Rebuild the evaluator to provide not only the decision of the result but also the explanation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc498085-6147-440c-9d62-08b7c9481725",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_user_message = \"\"\"You are evaluating a generated quiz based on the question bank that the assistant uses to create the quiz.\n",
    "  Here is the data:\n",
    "    [BEGIN DATA]\n",
    "    ************\n",
    "    [Question Bank]: {context}\n",
    "    ************\n",
    "    [Quiz]: {agent_response}\n",
    "    ************\n",
    "    [END DATA]\n",
    "\n",
    "## Examples of quiz questions\n",
    "Subject: <subject>\n",
    "   Categories: <category1>, <category2>\n",
    "   Facts:\n",
    "    - <fact 1>\n",
    "    - <fact 2>\n",
    "\n",
    "## Steps to make a decision\n",
    "Compare the content of the submission with the question bank using the following steps\n",
    "\n",
    "1. Review the question bank carefully. These are the only facts the quiz can reference\n",
    "2. Compare the information in the quiz to the question bank.\n",
    "3. Ignore differences in grammar or punctuation\n",
    "\n",
    "Remember, the quizzes should only include information from the question bank.\n",
    "\n",
    "\n",
    "## Additional rules\n",
    "- Output an explanation of whether the quiz only references information in the context.\n",
    "- Make the explanation brief only include a summary of your reasoning for the decsion.\n",
    "- Include a clear \"Yes\" or \"No\" as the first paragraph.\n",
    "- Reference facts from the quiz bank if the answer is yes\n",
    "\n",
    "Separate the decision and the explanation. For example:\n",
    "\n",
    "************\n",
    "Decision: <Y>\n",
    "************\n",
    "Explanation: <Explanation>\n",
    "************\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0be65c6c",
   "metadata": {},
   "source": [
    "Rebuild the check prompt template with the new prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6805362-eebc-4bd7-ac25-252ef3fcb739",
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_prompt = ChatPromptTemplate.from_messages([\n",
    "      (\"system\", eval_system_prompt),\n",
    "      (\"human\", eval_user_message),\n",
    "  ])\n",
    "eval_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f15d009",
   "metadata": {},
   "source": [
    "### Building multiple test cases to use the evaluator and new prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "251a1f9b",
   "metadata": {},
   "source": [
    "Define different test cases, prompts that include inputs and their expected responses.\n",
    "\n",
    "\n",
    "*_Note_*: Try adding your own inputs and responses!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8698fdd3-2912-4847-b3c9-57cff2cfd201",
   "metadata": {},
   "outputs": [],
   "source": [
    "# In a real application you would load your dataset from a file or logging tool.\n",
    "# Here we have a mix of examples with slightly different phrasing that our quiz application can support\n",
    "# and things we don't support. \n",
    "test_dataset = [\n",
    "  {\"input\": \"I'm trying to learn about science, can you give me a quiz to test my knowledge\",\n",
    "   \"response\": \"science\",\n",
    "   \"subjects\": [\"davinci\", \"telescope\", \"physics\", \"curie\"]},\n",
    "  {\"input\": \"I'm an geography expert, give a quiz to prove it?\",\n",
    "   \"response\": \"geography\",\n",
    "   \"subjects\": [\"paris\", \"france\", \"louvre\"]},\n",
    "   {\"input\": \"Quiz me about Italy\",\n",
    "   \"response\": \"geography\",\n",
    "   \"subjects\": [\"rome\", \"alps\", \"sicily\"]\n",
    "   },\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2427f7ff",
   "metadata": {},
   "source": [
    "Create funcion evaluate_dataset to look through the dataset and invoke the quiz generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "216c40cc-8e4b-4dd0-8a4f-a5d436bcdfcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_dataset(dataset, \n",
    "                     quiz_bank,\n",
    "                     assistant, \n",
    "                     evaluator):\n",
    "  eval_results = []\n",
    "  for row in dataset:\n",
    "    eval_result = {}\n",
    "    user_input = row[\"input\"] \n",
    "    answer = assistant.invoke({\"question\": user_input})\n",
    "    eval_response = evaluator.invoke({\"context\": quiz_bank, \"agent_response\": answer})\n",
    "    \n",
    "    eval_result[\"input\"] = user_input\n",
    "    eval_result[\"output\"] = answer\n",
    "    eval_result[\"grader_response\"] = eval_response\n",
    "    eval_results.append(eval_result)\n",
    "  return eval_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9209b7e2",
   "metadata": {},
   "source": [
    "Import functions to generate reports for the test_dataset evaluations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdb2c443-fdcf-4c8a-891c-00ca096ebfce",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from app import assistant_chain, quiz_bank\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "from langchain.prompts                import ChatPromptTemplate\n",
    "from langchain.chat_models            import ChatOpenAI\n",
    "from langchain.schema.output_parser   import StrOutputParser"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f6d73c7",
   "metadata": {},
   "source": [
    "Create the wrapper, incluide the LLM (in this case GTP-3.5-turbo)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "783b87f1-4322-4c47-be95-e8074a4b4655",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_eval_chain(prompt):\n",
    "  \n",
    "  return prompt | \\\n",
    "    ChatOpenAI(model=\"gpt-3.5-turbo\", \n",
    "               temperature=0) | \\\n",
    "    StrOutputParser()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9f66fc1",
   "metadata": {},
   "source": [
    "Use pandas to facilitate the creation of reports and define report_evals() function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a989e9c-5f00-406f-b5be-9d855225aab3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def report_evals(display_to_notebook=False):\n",
    "  assistant = assistant_chain()\n",
    "  model_graded_evaluator = create_eval_chain(eval_prompt)\n",
    "  eval_results = evaluate_dataset(test_dataset, \n",
    "                                  quiz_bank, \n",
    "                                  assistant, \n",
    "                                  model_graded_evaluator)\n",
    "  df = pd.DataFrame(eval_results)\n",
    "  ## clean up new lines to be html breaks\n",
    "  df_html = df.to_html().replace(\"\\\\n\",\"<br>\")\n",
    "  if display_to_notebook:\n",
    "    display(HTML(df_html))\n",
    "  else:\n",
    "    print(df_html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "879f8f46",
   "metadata": {},
   "source": [
    "Display the table with the results (grader_response).\n",
    "\n",
    "<p style=\"background-color:#ffc107; padding:15px; margin-left:20px\"> <b>Note:</b> Depending on the specific wording of the questions generated by your application, you may receive a different decision from the evaluation model in the following cell. This is a good example of why storing evaluation results for human review can be useful in refining model behavior.</p>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14b54bf8-af13-4aea-a6d4-bd964f930eb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "report_evals(display_to_notebook=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e557bff",
   "metadata": {},
   "source": [
    "## Integration to CircleCI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11fe73bb",
   "metadata": {},
   "source": [
    "Display the additional file for the CI/CD integrations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76150b2f-f1ca-4d59-908f-0133bd3a02d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cat save_eval_artifacts.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4932d945",
   "metadata": {},
   "source": [
    "Import the function trigger_eval_report adding our recent configurations and then at CircleCI website you will see the evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edd0f8d1-bd90-44e2-a7ff-4ea2f0bcab6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils import trigger_eval_report\n",
    "trigger_eval_report(course_repo, \n",
    "                    course_branch, \n",
    "                    [\"app.py\", \n",
    "                     \"save_eval_artifacts.py\",\n",
    "                     \"quiz_bank.txt\"], \n",
    "                    cci_api_key)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a019c64-b830-4020-8e9a-ea8c97f4fc22",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a37685-1c06-4f15-9770-76c6dc9c22ad",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c53ef803-ddd1-4849-813f-c2d959517f64",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d3b64ea-3b61-4800-8ce0-c08f092fb880",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cc96d50-ca40-44ac-acca-01aa78fd1cb9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1bbcbc0-6d23-42fd-900a-3c92228bfa95",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "203fb437-66ad-4b7c-a691-2f99487f78d0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0db21117-83e6-4e34-9553-20d2388668cb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40b96d79-96c5-44c8-b717-214fab45b05d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7fe53b19-6192-469d-9669-d173383d0d4c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}