exercise-handling-missing-values.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [Data Cleaning](https://www.kaggle.com/learn/data-cleaning) course.  You can reference the tutorial at [this link](https://www.kaggle.com/alexisbcook/handling-missing-values).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"In this exercise, you'll apply what you learned in the **Handling missing values** tutorial.\n\n# Setup\n\nThe questions below will give you feedback on your work. Run the following cell to set up the feedback system.","metadata":{}},{"cell_type":"code","source":"from learntools.core import binder\nbinder.bind(globals())\nfrom learntools.data_cleaning.ex1 import *\nprint(\"Setup Complete\")","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:22:48.963123Z","iopub.execute_input":"2022-07-14T11:22:48.964003Z","iopub.status.idle":"2022-07-14T11:22:52.883386Z","shell.execute_reply.started":"2022-07-14T11:22:48.963954Z","shell.execute_reply":"2022-07-14T11:22:52.882258Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1) Take a first look at the data\n\nRun the next code cell to load in the libraries and dataset you'll use to complete the exercise.","metadata":{}},{"cell_type":"code","source":"# modules we'll use\nimport pandas as pd\nimport numpy as np\n\n# read in all our data\nsf_permits = pd.read_csv(\"../input/building-permit-applications-data/Building_Permits.csv\")\n\n# set seed for reproducibility\nnp.random.seed(0) ","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:22:55.434565Z","iopub.execute_input":"2022-07-14T11:22:55.434988Z","iopub.status.idle":"2022-07-14T11:22:56.938683Z","shell.execute_reply.started":"2022-07-14T11:22:55.434919Z","shell.execute_reply":"2022-07-14T11:22:56.937487Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use the code cell below to print the first five rows of the `sf_permits` DataFrame.","metadata":{}},{"cell_type":"code","source":"# TODO: Your code here!\nsf_permits.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:23:32.562062Z","iopub.execute_input":"2022-07-14T11:23:32.562774Z","iopub.status.idle":"2022-07-14T11:23:32.600460Z","shell.execute_reply.started":"2022-07-14T11:23:32.562736Z","shell.execute_reply":"2022-07-14T11:23:32.599412Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Does the dataset have any missing values?  Once you have an answer, run the code cell below to get credit for your work.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq1.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:23:39.814235Z","iopub.execute_input":"2022-07-14T11:23:39.814837Z","iopub.status.idle":"2022-07-14T11:23:39.822989Z","shell.execute_reply.started":"2022-07-14T11:23:39.814804Z","shell.execute_reply":"2022-07-14T11:23:39.821804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Line below will give you a hint\n#q1.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2) How many missing data points do we have?\n\nWhat percentage of the values in the dataset are missing?  Your answer should be a number between 0 and 100.  (If 1/4 of the values in the dataset are missing, the answer is 25.)","metadata":{}},{"cell_type":"code","source":"missing_value_count=sf_permits.isnull().sum()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:30:18.877647Z","iopub.execute_input":"2022-07-14T11:30:18.878027Z","iopub.status.idle":"2022-07-14T11:30:19.106177Z","shell.execute_reply.started":"2022-07-14T11:30:18.877997Z","shell.execute_reply":"2022-07-14T11:30:19.105245Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TODO: Your code here!\n\ntotal_row=np.product(sf_permits.shape)\ntotal_missing=missing_value_count.sum()\npercent_missing =(total_missing/total_row)*100\nprint(percent_missing)\n\n# Check your answer\nq2.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T11:30:53.176922Z","iopub.execute_input":"2022-07-14T11:30:53.177322Z","iopub.status.idle":"2022-07-14T11:30:53.187327Z","shell.execute_reply.started":"2022-07-14T11:30:53.177292Z","shell.execute_reply":"2022-07-14T11:30:53.186364Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q2.hint()\n#q2.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3) Figure out why the data is missing\n\nLook at the columns **\"Street Number Suffix\"** and **\"Zipcode\"** from the [San Francisco Building Permits dataset](https://www.kaggle.com/aparnashastry/building-permit-applications-data). Both of these contain missing values. \n- Which, if either, are missing because they don't exist? \n- Which, if either, are missing because they weren't recorded?  \n\nOnce you have an answer, run the code cell below.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq3.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:18:59.230390Z","iopub.execute_input":"2022-07-14T12:18:59.231155Z","iopub.status.idle":"2022-07-14T12:18:59.244591Z","shell.execute_reply.started":"2022-07-14T12:18:59.231065Z","shell.execute_reply":"2022-07-14T12:18:59.243711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Line below will give you a hint\n#q3.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4) Drop missing values: rows\n\nIf you removed all of the rows of `sf_permits` with missing values, how many rows are left?\n\n**Note**: Do not change the value of `sf_permits` when checking this.  ","metadata":{}},{"cell_type":"code","source":"columns_with_na_dropped = sf_permits.dropna(axis=1)\n","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:22:09.645904Z","iopub.execute_input":"2022-07-14T12:22:09.646636Z","iopub.status.idle":"2022-07-14T12:22:09.920173Z","shell.execute_reply.started":"2022-07-14T12:22:09.646601Z","shell.execute_reply":"2022-07-14T12:22:09.918906Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TODO: Your code here!\nprint(\"Columns in original dataset: %d \\n\" % sf_permits.shape[1])\nprint(\"Columns with na's dropped: %d\" % columns_with_na_dropped.shape[1])\n","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:22:31.505548Z","iopub.execute_input":"2022-07-14T12:22:31.505907Z","iopub.status.idle":"2022-07-14T12:22:31.512840Z","shell.execute_reply.started":"2022-07-14T12:22:31.505879Z","shell.execute_reply":"2022-07-14T12:22:31.511591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Once you have an answer, run the code cell below.","metadata":{}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq4.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:22:35.281663Z","iopub.execute_input":"2022-07-14T12:22:35.282117Z","iopub.status.idle":"2022-07-14T12:22:35.290206Z","shell.execute_reply.started":"2022-07-14T12:22:35.282081Z","shell.execute_reply":"2022-07-14T12:22:35.289295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Line below will give you a hint\n#q4.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5) Drop missing values: columns\n\nNow try removing all the columns with empty values.  \n- Create a new DataFrame called `sf_permits_with_na_dropped` that has all of the columns with empty values removed.  \n- How many columns were removed from the original `sf_permits` DataFrame? Use this number to set the value of the `dropped_columns` variable below.","metadata":{}},{"cell_type":"code","source":"# TODO: Your code here\nsf_permits_with_na_dropped = sf_permits.dropna(axis=1)\noriginal=sf_permits.shape[1]\ndropped=sf_permits_with_na_dropped.shape[1]\n\n\n\ndropped_columns = original-dropped\n\n# Check your answer\nq5.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:29:19.801164Z","iopub.execute_input":"2022-07-14T12:29:19.801520Z","iopub.status.idle":"2022-07-14T12:29:20.075602Z","shell.execute_reply.started":"2022-07-14T12:29:19.801493Z","shell.execute_reply":"2022-07-14T12:29:20.074366Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q5.hint()\n#q5.solution()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:29:30.045333Z","iopub.execute_input":"2022-07-14T12:29:30.045704Z","iopub.status.idle":"2022-07-14T12:29:30.050834Z","shell.execute_reply.started":"2022-07-14T12:29:30.045675Z","shell.execute_reply":"2022-07-14T12:29:30.049597Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 6) Fill in missing values automatically\n\nTry replacing all the NaN's in the `sf_permits` data with the one that comes directly after it and then replacing any remaining NaN's with 0.  Set the result to a new DataFrame `sf_permits_with_na_imputed`.","metadata":{}},{"cell_type":"code","source":"# TODO: Your code here\nsf_permits_with_na_imputed =sf_permits.fillna(method='bfill', axis=0).fillna(0)\n\n# Check your answer\nq6.check()","metadata":{"execution":{"iopub.status.busy":"2022-07-14T12:32:55.542034Z","iopub.execute_input":"2022-07-14T12:32:55.542414Z","iopub.status.idle":"2022-07-14T12:32:56.557540Z","shell.execute_reply.started":"2022-07-14T12:32:55.542384Z","shell.execute_reply":"2022-07-14T12:32:56.556773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lines below will give you a hint or solution code\n#q6.hint()\n#q6.solution()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# More practice\n\nIf you're looking for more practice handling missing values:\n\n* Check out [this noteboook](https://www.kaggle.com/alexisbcook/missing-values) on handling missing values using scikit-learn's imputer. \n* Look back at the \"Zipcode\" column in the `sf_permits` dataset, which has some missing values. How would you go about figuring out what the actual zipcode of each address should be? (You might try using another dataset. You can search for datasets about San Fransisco on the [Datasets listing](https://www.kaggle.com/datasets).) \n\n# Keep going\n\nIn the next lesson, learn how to [**apply scaling and normalization**](https://www.kaggle.com/alexisbcook/scaling-and-normalization) to transform your data.","metadata":{}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/data-cleaning/discussion) to chat with other learners.*","metadata":{}}]}