python-datascience/05.04-Feature-Engineering.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Feature Engineering"},{"metadata":{},"cell_type":"markdown","source":"The previous sections outline the fundamental ideas of machine learning, but all of the examples assume that you have numerical data in a tidy, ``[n_samples, n_features]`` format.\nIn the real world, data rarely comes in such a form.\nWith this in mind, one of the more important steps in using machine learning in practice is *feature engineering*: that is, taking whatever information you have about your problem and turning it into numbers that you can use to build your feature matrix.\n\nIn this section, we will cover a few common examples of feature engineering tasks: features for representing *categorical data*, features for representing *text*, and features for representing *images*.\nAdditionally, we will discuss *derived features* for increasing model complexity and *imputation* of missing data.\nOften this process is known as *vectorization*, as it involves converting arbitrary data into well-behaved vectors."},{"metadata":{},"cell_type":"markdown","source":"## Categorical Features\n\nOne common type of non-numerical data is *categorical* data.\nFor example, imagine you are exploring some data on housing prices, and along with numerical features like \"price\" and \"rooms\", you also have \"neighborhood\" information.\nFor example, your data might look something like this:"},{"metadata":{"trusted":false},"cell_type":"code","source":"data = [\n    {'price': 850000, 'rooms': 4, 'neighborhood': 'Queen Anne'},\n    {'price': 700000, 'rooms': 3, 'neighborhood': 'Fremont'},\n    {'price': 650000, 'rooms': 3, 'neighborhood': 'Wallingford'},\n    {'price': 600000, 'rooms': 2, 'neighborhood': 'Fremont'}\n]","execution_count":1,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You might be tempted to encode this data with a straightforward numerical mapping:"},{"metadata":{"trusted":false},"cell_type":"code","source":"{'Queen Anne': 1, 'Fremont': 2, 'Wallingford': 3};","execution_count":2,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It turns out that this is not generally a useful approach in Scikit-Learn: the package's models make the fundamental assumption that numerical features reflect algebraic quantities.\nThus such a mapping would imply, for example, that *Queen Anne < Fremont < Wallingford*, or even that *Wallingford - Queen Anne = Fremont*, which (niche demographic jokes aside) does not make much sense.\n\nIn this case, one proven technique is to use *one-hot encoding*, which effectively creates extra columns indicating the presence or absence of a category with a value of 1 or 0, respectively.\nWhen your data comes as a list of dictionaries, Scikit-Learn's ``DictVectorizer`` will do this for you:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.feature_extraction import DictVectorizer\nvec = DictVectorizer(sparse=False, dtype=int)\nvec.fit_transform(data)","execution_count":3,"outputs":[{"data":{"text/plain":"array([[     0,      1,      0, 850000,      4],\n       [     1,      0,      0, 700000,      3],\n       [     0,      0,      1, 650000,      3],\n       [     1,      0,      0, 600000,      2]], dtype=int64)"},"execution_count":3,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"Notice that the 'neighborhood' column has been expanded into three separate columns, representing the three neighborhood labels, and that each row has a 1 in the column associated with its neighborhood.\nWith these categorical features thus encoded, you can proceed as normal with fitting a Scikit-Learn model.\n\nTo see the meaning of each column, you can inspect the feature names:"},{"metadata":{"trusted":false},"cell_type":"code","source":"vec.get_feature_names()","execution_count":4,"outputs":[{"data":{"text/plain":"['neighborhood=Fremont',\n 'neighborhood=Queen Anne',\n 'neighborhood=Wallingford',\n 'price',\n 'rooms']"},"execution_count":4,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"There is one clear disadvantage of this approach: if your category has many possible values, this can *greatly* increase the size of your dataset.\nHowever, because the encoded data contains mostly zeros, a sparse output can be a very efficient solution:"},{"metadata":{"trusted":false},"cell_type":"code","source":"vec = DictVectorizer(sparse=True, dtype=int)\nvec.fit_transform(data)","execution_count":5,"outputs":[{"data":{"text/plain":"<4x5 sparse matrix of type '<class 'numpy.int64'>'\n\twith 12 stored elements in Compressed Sparse Row format>"},"execution_count":5,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"Many (though not yet all) of the Scikit-Learn estimators accept such sparse inputs when fitting and evaluating models. ``sklearn.preprocessing.OneHotEncoder`` and ``sklearn.feature_extraction.FeatureHasher`` are two additional tools that Scikit-Learn includes to support this type of encoding."},{"metadata":{},"cell_type":"markdown","source":"## Text Features\n\nAnother common need in feature engineering is to convert text to a set of representative numerical values.\nFor example, most automatic mining of social media data relies on some form of encoding the text as numbers.\nOne of the simplest methods of encoding data is by *word counts*: you take each snippet of text, count the occurrences of each word within it, and put the results in a table.\n\nFor example, consider the following set of three phrases:"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"sample = ['problem of evil',\n          'evil queen',\n          'horizon problem']","execution_count":6,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For a vectorization of this data based on word count, we could construct a column representing the word \"problem,\" the word \"evil,\" the word \"horizon,\" and so on.\nWhile doing this by hand would be possible, the tedium can be avoided by using Scikit-Learn's ``CountVectorizer``:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.feature_extraction.text import CountVectorizer\n\nvec = CountVectorizer()\nX = vec.fit_transform(sample)\nX","execution_count":7,"outputs":[{"data":{"text/plain":"<3x5 sparse matrix of type '<class 'numpy.int64'>'\n\twith 7 stored elements in Compressed Sparse Row format>"},"execution_count":7,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"The result is a sparse matrix recording the number of times each word appears; it is easier to inspect if we convert this to a ``DataFrame`` with labeled columns:"},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\npd.DataFrame(X.toarray(), columns=vec.get_feature_names())","execution_count":8,"outputs":[{"data":{"text/html":"<div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>evil</th>\n      <th>horizon</th>\n      <th>of</th>\n      <th>problem</th>\n      <th>queen</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"   evil  horizon  of  problem  queen\n0     1        0   1        1      0\n1     1        0   0        0      1\n2     0        1   0        1      0"},"execution_count":8,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"There are some issues with this approach, however: the raw word counts lead to features which put too much weight on words that appear very frequently, and this can be sub-optimal in some classification algorithms.\nOne approach to fix this is known as *term frequency-inverse document frequency* (*TF–IDF*) which weights the word counts by a measure of how often they appear in the documents.\nThe syntax for computing these features is similar to the previous example:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.feature_extraction.text import TfidfVectorizer\nvec = TfidfVectorizer()\nX = vec.fit_transform(sample)\npd.DataFrame(X.toarray(), columns=vec.get_feature_names())","execution_count":9,"outputs":[{"data":{"text/html":"<div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>evil</th>\n      <th>horizon</th>\n      <th>of</th>\n      <th>problem</th>\n      <th>queen</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0.517856</td>\n      <td>0.000000</td>\n      <td>0.680919</td>\n      <td>0.517856</td>\n      <td>0.000000</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0.605349</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>0.795961</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.000000</td>\n      <td>0.795961</td>\n      <td>0.000000</td>\n      <td>0.605349</td>\n      <td>0.000000</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"       evil   horizon        of   problem     queen\n0  0.517856  0.000000  0.680919  0.517856  0.000000\n1  0.605349  0.000000  0.000000  0.000000  0.795961\n2  0.000000  0.795961  0.000000  0.605349  0.000000"},"execution_count":9,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"For an example of using TF-IDF in a classification problem, see [In Depth: Naive Bayes Classification](05.05-Naive-Bayes.ipynb)."},{"metadata":{},"cell_type":"markdown","source":"## Image Features\n\nAnother common need is to suitably encode *images* for machine learning analysis.\nThe simplest approach is what we used for the digits data in [Introducing Scikit-Learn](05.02-Introducing-Scikit-Learn.ipynb): simply using the pixel values themselves.\nBut depending on the application, such approaches may not be optimal.\n\nA comprehensive summary of feature extraction techniques for images is well beyond the scope of this section, but you can find excellent implementations of many of the standard approaches in the [Scikit-Image project](http://scikit-image.org).\nFor one example of using Scikit-Learn and Scikit-Image together, see [Feature Engineering: Working with Images](05.14-Image-Features.ipynb)."},{"metadata":{},"cell_type":"markdown","source":"## Derived Features\n\nAnother useful type of feature is one that is mathematically derived from some input features.\nWe saw an example of this in [Hyperparameters and Model Validation](05.03-Hyperparameters-and-Model-Validation.ipynb) when we constructed *polynomial features* from our input data.\nWe saw that we could convert a linear regression into a polynomial regression not by changing the model, but by transforming the input!\nThis is sometimes known as *basis function regression*, and is explored further in [In Depth: Linear Regression](05.06-Linear-Regression.ipynb).\n\nFor example, this data clearly cannot be well described by a straight line:"},{"metadata":{"trusted":false},"cell_type":"code","source":"%matplotlib inline\nimport numpy as np\nimport matplotlib.pyplot as plt\n\nx = np.array([1, 2, 3, 4, 5])\ny = np.array([4, 2, 1, 3, 7])\nplt.scatter(x, y);","execution_count":10,"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAEACAYAAACatzzfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD1JJREFUeJzt3X+M5PVdx/HX67q7OAcuJnYCthdmSxuyQbuWawKN1Pjd\n0G1pG+GvRqqmCX80IalCGmOKJOaGP2riH1ZJ7P1BmCI1MG64gFZTCRjua4OasuVHD7g7ayTfBQpc\nvo2IIhvTyts/5nvXu+X25jvszH4/OzwfyYXZuc9+9z1w97zPfWaWcUQIAJCuPU0PAAA4N0INAIkj\n1ACQOEINAIkj1ACQOEINAImrFWrbX7b9rO0jtu+1PTfpwQAAA0NDbft9kn5X0v6IWJI0I+mGSQ8G\nABiYqbnuPZLOt/2WpL2SXp7cSACA0w3dUUfEy5L+RNILkn4o6T8j4h8mPRgAYKDO0cfPSbpeUkfS\n+yRdYPs3Jz0YAGCgztHHJyQ9HxH/IUm2H5D0K5LuO32Rbf6nIQAwoojwsDV1XvXxgqSP2f4Z25Z0\njaRjW3zBqfxx4MCBxmfg8fH4eHzT96OuOmfUj0s6JOkpSd+XZEl31v4KAIBtqfWqj4i4XdLtE54F\nAHAWfGdiDVmWNT3CRPH4djce3/TzKOck57yQHeO6FgC8G9hWjOnJRABAgwg1ACSOUANA4gg1ACSO\nUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA\n4oaG2vZltp+y/WT1z9dt37wTwwEARnwrLtt7JL0k6aqIeHHTz/FWXAAwgkm9FdcnJP375kgDwCSU\nZam1tTWVZdn0KI0aNdS/Iak/iUEA4HT9/qo6nUWtrNykTmdR/f5q0yM1pvbRh+1ZSS9Lujwi3vbH\nG0cfAMalLEt1Oova2DgsaUnSEbVay1pfP652u930eGNT9+hjZoRrflrSE2eL9EndbvfU7SzLlGXZ\nCJcHgIGiKDQ3t6CNjaXqniXNznZUFMWuDnWe58rzfOTPG2VH3Zf0UETcs8XPs6MGMBbsqM9U64za\n9l4Nnkh8YLuDAcAw7XZbvd5BtVrLmp/fr1ZrWb3ewamK9ChGenneOS/EjhrAmJVlqaIotLCwMJWR\nrrujJtQA0JBJvY4aALDDCDUAJI5QA0DiCDUAJI5QA0DiCDUAJI5QA0DiCDUAJI5QA0DiCDUAJI5Q\nA0DiCDUAJI5QA0DiCDUAJI5QA0DiCDUAJI5QA0DiCDUAJK7um9teaPt+28dsP2f7qkkPBgAYmKm5\n7g5J346Iz9mekbR3gjMBAE4z9M1tbc9LeioiPjhkHW9uCwAjGOeb235A0o9s3237Sdt32m5tf0QA\nQB11jj5mJO2X9KWI+J7tP5N0q6QDmxd2u91Tt7MsU5Zl45kSAKZAnufK83zkz6tz9HGRpH+JiEur\njz8u6SsR8eub1nH0AQAjGNvRR0SckPSi7cuqu66RdHSb8wEAahq6o5Yk278s6S5Js5Kel3RjRLy+\naQ07agAYQd0dda1Q1/yChBoARjDOV30AABpEqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEg\ncYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABI3U2eR7ULS\n65LekvTjiLhykkMBAH6qVqg1CHQWEa9NchgAwNvVPfrwCGsBAGNUN74h6RHba7a/OMmBAABnqnv0\ncXVEvGK7rUGwj0XEY5sXdbvdU7ezLFOWZWMZEgCmQZ7nyvN85M9zRIz2CfYBSf8dEV/bdH+Mei0A\neDezrYjwsHVDjz5s77V9QXX7fEmflPTs9kcEANRR5+jjIkkP2o5q/b0R8fBkxwIAnDTy0ceWF+Lo\nAwBGMrajDwBAswg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1\nACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4mqH2vYe20/a/tYkBwIAnGmUHfUtko5O\napBUlWWptbU1lWXZ9CgA3qVqhdr2PkmfkXTXZMdJS7+/qk5nUSsrN6nTWVS/v9r0SADehRwRwxfZ\n90v6qqQLJf1eRFx3ljVR51q7RVmW6nQWtbFxWNKSpCNqtZa1vn5c7Xa76fEATAHbiggPWzdT40Kf\nlXQiIp62nUna8qLdbvfU7SzLlGVZnVmTVBSF5uYWtLGxVN2zpNnZjoqiINQA3pE8z5Xn+cifN3RH\nbfuPJP22pJ9Iakn6WUkPRMQXNq1jRw0AI6i7ox56Rh0Rt0XEJRFxqaQbJD26OdLTqN1uq9c7qFZr\nWfPz+9VqLavXO0ikAey4WmfUpxbbv6Z3yRn1SWVZqigKLSwsEGkAY1V3Rz1SqId8wakMNQBMytiO\nPgAAzSLUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0A\niSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiZsZtsD2eZK+I2muWn8oIm6f9GAAgIGhO+qI+F9J\nyxFxhaSPSPq07SsnPhmAocqy1NramsqybHoUTFCto4+IeLO6eZ4Gu2rebhxoWL+/qk5nUSsrN6nT\nWVS/v9r0SJgQRwxvru09kp6Q9EFJX4+IPzjLmqhzLQDbV5alOp1FbWwclrQk6YharWWtrx9Xu91u\nejzUZFsR4WHrhp5RS1JEvCXpCtvzkv7a9uURcXTzum63e+p2lmXKsqz2wADqK4pCc3ML2thYqu5Z\n0uxsR0VREOqE5XmuPM9H/rxaO+ozPsH+Q0n/ExFf23Q/O2pgh7Cjng51d9RDz6htv9f2hdXtlqQV\nSce3PyKAd6rdbqvXO6hWa1nz8/vVai2r1ztIpKfU0B217Q9LukeDqO+RtBoRXz3LOnbUwA4ry1JF\nUWhhYYFI70J1d9QjH32c4wsSagAYwdiOPgAAzSLUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPU\nAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiSPUAJA4Qg0AiavzLuT7\nbD9q+znbz9i+eScGAwAM1HkX8oslXRwRT9u+QNITkq6PiOOb1vHmtrsQ72INNGdsb24bEa9GxNPV\n7TckHZP0/u2PiKb1+6vqdBa1snKTOp1F9furTY8E4CyG7qjPWGwvSMol/VIV7dN/jh31LlKWpTqd\nRW1sHJa0JOmIWq1lra8fZ2cN7JC6O+qZES54gaRDkm7ZHOmTut3uqdtZlinLsrqXxw4rikJzcwva\n2Fiq7lnS7GxHRVEQamBC8jxXnucjf16tHbXtGUl/J+nvI+KOLdawo95F2FEDzRvbGXXlG5KObhVp\n7D7tdlu93kG1Wsuan9+vVmtZvd5BIg0kqM6rPq6W9B1Jz0iK6sdtEfHQpnXsqHchXvUBNKfujnqk\nJxOHfEFCDQAjGPfRBwCgIYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJH\nqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABJHqAEgcYQaABI3NNS2e7ZP2D6yEwMBAM5UZ0d9t6RP\nTXoQAMDZDQ11RDwm6bUdmAUYu7Istba2prIsmx4FeMc4o8bU6vdX1eksamXlJnU6i+r3V5seCXhH\nHBHDF9kdSX8bEUvnWBN1rgXshLIs1eksamPjsKQlSUfUai1rff242u120+MBkiTbiggPWzczzi/a\n7XZP3c6yTFmWjfPyQG1FUWhubkEbGyf3Fkuane2oKApCjcbkea48z0f+vLo76gUNdtQfPscadtRI\nBjtq7AZ1d9R1Xp53n6R/lnSZ7Rds3ziOAYFJarfb6vUOqtVa1vz8frVay+r1DhJp7Eq1dtS1LsSO\nGgkqy1JFUWhhYYFIIzl1d9SEGgAaMrajDwBAswg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1\nACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSOUANA4gg1ACSuVqhtX2v7uO0f\n2P7KpIcCAPxUnXch3yPpzyV9StIvSvq87cVJD5aSPM+bHmGieHy7G49v+tXZUV8p6d8iYj0ifizp\nryRdP9mx0jLtv1B4fLsbj2/61Qn1+yW9eNrHL1X3AQB2AE8mAkDiHBHnXmB/TFI3Iq6tPr5VUkTE\nH29ad+4LAQDeJiI8bE2dUL9H0r9KukbSK5Iel/T5iDg2jiEBAOc2M2xBRPyf7d+R9LAGRyU9Ig0A\nO2fojhoA0KxtP5k4zd8MY7tn+4TtI03PMgm299l+1PZztp+xfXPTM42T7fNsf9f2U9XjO9D0TONm\ne4/tJ21/q+lZxs12Yfv71X+/x5ueZ9xsX2j7ftvHqt+DV225djs76uqbYX6gwfn1y5LWJN0QEcff\n8UUTYvvjkt6Q9M2IWGp6nnGzfbGkiyPiadsXSHpC0vXT8t9PkmzvjYg3q+da/knSzRExNb/pbX9Z\n0kclzUfEdU3PM062n5f00Yh4relZJsH2X0j6x4i42/aMpL0R8V9nW7vdHfVUfzNMRDwmaSp/kUhS\nRLwaEU9Xt9+QdExT9hr5iHizunmeBs/JTM1Zn+19kj4j6a6mZ5kQa0pfQmx7XtKvRsTdkhQRP9kq\n0tL2/yXwzTBTwvaCpI9I+m6zk4xXdTTwlKRXJT0SEWtNzzRGfyrp9zVFf/hsEpIesb1m+4tNDzNm\nH5D0I9t3V0dXd9pubbV4Kv+0wmiqY49Dkm6pdtZTIyLeiogrJO2TdJXty5ueaRxsf1bSiepvRK5+\nTJurI2K/Bn9r+FJ1FDktZiTtl/T16jG+KenWrRZvN9Q/lHTJaR/vq+7DLlGdjR2S9JcR8TdNzzMp\n1V8rD0u6tulZxuRqSddV57h9Scu2v9nwTGMVEa9U/ywlPajBUeu0eEnSixHxverjQxqE+6y2G+o1\nSR+y3bE9J+kGSdP27PO07lZO+oakoxFxR9ODjJvt99q+sLrdkrQiaSqeKI2I2yLikoi4VIPfd49G\nxBeanmtcbO+t/qYn2+dL+qSkZ5udanwi4oSkF21fVt11jaSjW60f+g0vQ77YVH8zjO37JGWSft72\nC5IOnDz8nwa2r5b0W5Keqc5xQ9JtEfFQs5ONzS9Iuqd6ddIeSasR8e2GZ0I9F0l6sPpfU8xIujci\nHm54pnG7WdK9tmclPS/pxq0W8g0vAJA4nkwEgMQRagBIHKEGgMQRagBIHKEGgMQRagBIHKEGgMQR\nagBI3P8DWfnC9JC1xVsAAAAASUVORK5CYII=\n","text/plain":"<matplotlib.figure.Figure at 0x114836550>"},"metadata":{},"output_type":"display_data"}]},{"metadata":{},"cell_type":"markdown","source":"Still, we can fit a line to the data using ``LinearRegression`` and get the optimal result:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\nX = x[:, np.newaxis]\nmodel = LinearRegression().fit(X, y)\nyfit = model.predict(X)\nplt.scatter(x, y)\nplt.plot(x, yfit);","execution_count":11,"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAEACAYAAACatzzfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEmtJREFUeJzt3X+M5HV9x/Hn+7w9ugc9NHUDInFWMeRi67YeiRix6Zx4\nakEhMR7eXRvRIAmpLcY0jdakYf3Dpv2jtjb1JNaVasONJwSEViooMDXiD04EAYFahFlQlAypPbGs\nCSfv/jFzcCx7t9/hZvb72dnnI9nc7OxnZt9zP1732ff3Pd9vZCaSpHKtq7sASdKRGdSSVDiDWpIK\nZ1BLUuEMakkqnEEtSYWrFNQR8cGIuDsi7oyIyyNiw6gLkyT1LBvUEXES8GfAlsycAdYDO0ZdmCSp\nZ33FdS8Ajo2Ip4CNwCOjK0mSdKhld9SZ+Qjwd8BDwE+A/83Mr426MElST5XWxwuBc4EGcBJwXETs\nGnVhkqSeKq2PNwEPZOb/AETEVcDrgT2HLooITxoiSQPKzFhuTZWpj4eA10XEb0REAGcC9x7mG47l\nxyWXXFJ7Db4+X5+vb/w+qqrSo74VuBK4Hfg+EMCnK38HSdJRqTT1kZkfBT464lokSUvwnYkVNJvN\nuksYKV/f6ubrG38xSJ/kiE8UkcN6LklaCyKCHNLBRElSjQxqSSqcQS1JhTOoJalwBrUkFc6glqTC\nGdSSVDiDWpIKZ1BLUuEMakkqnEEtSYUzqCWpcAa1JBXOoJakwhnUklQ4g1qSCmdQS1Lhlg3qiDg1\nIm6PiO/1f90fERevRHGSpAEvxRUR64AfA6dn5sOLvualuCRpAKO6FNebgB8tDmlJGoVut8u+ffvo\ndrt1l1KrQYP6XUBrFIVI0qFarb00GpvZtu0iGo3NtFp76y6pNpVbHxExATwCvCozn/Pfm60PScPS\n7XZpNDazsHAzMAPcyeTkVubn72Nqaqru8oamautj/QDP+YfAbUuF9EGzs7NP3242mzSbzQGeXpJ6\nOp0OGzZMs7Aw079nhomJBp1OZ1UHdbvdpt1uD/y4QXbULeArmfm5w3zdHbWkoXBH/WyVetQRsZHe\ngcSrjrYwSVrO1NQUc3O7mZzcyqZNW5ic3Mrc3O6xCulBDDSed8Qnckctaci63S6dTofp6emxDOmq\nO2qDWpJqMqo5aknSCjOoJalwBrUkFc6glqTCGdSSVDiDWpIKZ1BLUuEMakkqnEEtSYUzqCWpcAa1\nJBXOoJakwhnUklQ4g1qSCmdQS1LhDGpJKpxBLUmFM6glqXBVL257fERcERH3RsQPIuL0URcmSepZ\nX3HdJ4DrMnN7RKwHNo6wJknSIZa9uG1EbAJuz8xTllnnxW0laQDDvLjty4HHIuKyiPheRHw6IiaP\nvkRJUhVVWh/rgS3A+zPzuxHxD8CHgUsWL5ydnX36drPZpNlsDqdKSRoD7Xabdrs98OOqtD5OAL6V\nma/of/4G4EOZ+fZF62x9SNIAhtb6yMxHgYcj4tT+XWcC9xxlfZKkipbdUQNExO8CnwEmgAeA92bm\n/kVr3FFL0gCq7qgrBXXFb2hQS9IAhjn1IUmqkUEtSYUzqCWpcAa1JBXOoJakwhnUklQ4g1qSCmdQ\nS1LhDGpJKpxBLUmFM6glqXAGtSQVzqCWpMIZ1JJUOINakgpnUEtS4QxqSSqcQS1JhTOoJalw66ss\niogOsB94CngyM187yqIkSc+oFNT0ArqZmT8fZTGSpOeq2vqIAdZKkoaoavgm8NWI2BcRF46yIEnS\ns1VtfZyRmT+NiCl6gX1vZn5j8aLZ2dmnbzebTZrN5lCKlKRx0G63abfbAz8uMnOwB0RcAjyemR9f\ndH8O+lyStJZFBJkZy61btvURERsj4rj+7WOBNwN3H32JkqQqqrQ+TgCujojsr788M28YbVmSpIMG\nbn0c9olsfUjSQIbW+pCklXbgAFx/PVx4IfzqV3VXU7+qUx+SNFKZ8K1vwZ49cMUVMD0Nu3bBr39d\nd2X1M6gl1equu3rh3GrBxo29cL7lFnjlK+uurBwGtaQV9+CDvWDeswcefxx27IBrroGZGYhlO7Zr\njwcTJa2IRx+FL36xF84/+hG885293fPrXw/r1ujRsqoHEw1qSSOzfz9cdVVv93zrrfD2t/fC+U1v\ngomJuqurn0EtqRYLC/DlL/fC+Wtfgze+EXbuhLe9rdeD1jMMakkr5sABuPHGXlvj2mvhtNN6O+d3\nvANe+MK6qyuXQS1ppA43TnfeefCSl9Rd3epQNaid+pA0EMfpVp5BLWlZh47T/eIXvZ6z43Qrx9aH\npCU5Tjd69qglDcxxupVlUEuqxHG6+hjUkg5rqXG6nTt743QvelHd1a0dBrWkZ3GcrjyO50kCHKcb\nBwa1NIYcpxsvtj6kMXHoON3998P27Y7TlW7oPeqIWAd8F/hxZp6zxNcNammFOU63uo2iR/0B4B5g\n0/OuahXqdrt0Oh2mp6eZmpqquxzpOeN0W7fC+94HX/qS43TjqtIPRBFxMnAW8JnRllOWVmsvjcZm\ntm27iEZjM63W3rpL0hp18GKv558PJ50El14KZ50FnU4voM87z5AeZ5VaHxFxBfAx4Hjgz9dC66Pb\n7dJobGZh4WZgBriTycmtzM/f585aK8JxuvE3tNZHRJwNPJqZd0REEzjsk87Ozj59u9ls0mw2q9Ra\npE6nw4YN0ywszPTvmWFiokGn0zGoNVKO042vdrtNu90e+HHL7qgj4q+BPwYOAJPAbwJXZea7F61z\nRy09T0uN0+3a5TjduBvJOxMj4g9YI60P6PWoL7jgT5iYaPDkk/PMze1m58531V2WxoRnp5NBPSRO\nfWiYHKfToTzXh1QIz06nwzGopRp5sVdVYVBLK8xxOg3Ks+dJK8RxOo2aQS09D4eO0z3+OOzY4dnp\nNDq2PqSKFo/Tbd/eOyjoOJ2eL3vU0hAsHqc755xeODtOp2EwqKXnaalxul274OyzHafTcBnU0gAO\njtO1Wr1es+N0WgkGtbSMxeN0L395r63hOJ1WiuN50mEcHKf7whdgcrK3c/7mN+GUU+quTFqaQa01\nYfE43c6dvRPuO06n1cDWh8aW43QqnT1qrUn798PVV/fCed++3tnpHKdTqQxqrRmO02m1Mqg11hyn\n0zgwqDV2Do7TtVq93rPjdFrtHM/T2HCcTmudQa0iHRyna7Weudir43Raq2x9qBiO02mtGVqPOiKO\nAb4ObKC3A78yMz+6xDqDWgNbapxu1y4480zH6TT+hnowMSI2ZuYTEfEC4Bbg4sy8ddEag1qVLCzA\nddf1wtlxuqPT7XbpdDpMT08zNTVVdzkaUNWgrvQDZWY+0b95DL1dtYmsgRw4ADfcAO95D5x0Enzq\nU71gnp/v7ai3bzekB9Vq7aXR2My2bRfRaGym1dpbd0kakao76nXAbcApwCcz8y+XWOOOWs/iON3o\ndLtdGo3NLCzcDMwAdzI5uZX5+fvcWa8iQx3Py8yngNdExCbgSxHxqsy8Z/G62dnZp283m02azWbl\ngjU+7rrrmYkNx+lGo9PpsGHDNAsLM/17ZpiYaNDpdAzqgrXbbdrt9sCPG3jqIyL+Cvi/zPz4ovvd\nUa9hDz7Ym3Pes+eZcbqdOx2nGxV31ONhaDvqiHgx8GRm7o+ISWAb8DdDqFGr3MFxulYL7r+/12f+\n1Kccp1sJU1NTzM3t5oILtjIx0eDJJ+eZm9ttSI+pKuN5rwY+R+/A4zpgb2Z+bIl17qjXAM9OVxan\nPlY3z/WhofnVr3pnp3OcThoug1pH5cABuOmmXjhfey1s2eLZ6aRhM6g1sEz49rd74ew4nTR6nj1P\nld19dy+cHaeTymRQr1GHjtPt3+/Z6aSS2fpYQ5Yap/PsdFJ97FELcJxOKplBvYZ5djppdTCo15hD\nx+m82Ku0OhjUa4Bnp5NWN8fzxpgXe5XWFoN6lVh8sdcdOxynk9YKWx8F82Kv0nizR71KHTpOd+ut\ncM45jtNJ48qgXkUWFnpnp2u1HKeT1hKDunAHDsCNNz5zdjrH6aS1x6Au0MFxuj174IorHKeT1jrH\n8wpycJyu1eq1MhynkzQIg3pEDo7T7dkDjz/e2zlfc43jdJIGZ+tjiBynkzSIofWoI+Jk4PPACcBT\nwD9n5j8usW5NBvX+/XDVVb3ds+N0kgYxzKA+ETgxM++IiOOA24BzM/O+RevWTFCP0zidV7GW6lM1\nqJf9gTwzf5aZd/Rv/xK4F3jp0Ze4uhw4ANdfD+efDyedBJde2gvm+fneG1S2b199Id1q7aXR2My2\nbRfRaGym1dpbd0mSljBQjzoipoE28Dv90D70a2O5o37sMZidHb9xum63S6OxmYWFm4EZ4E4mJ7cy\nP3+fO2tphQx9PK/f9rgS+MDikD5odnb26dvNZpNms1n16Yt17LG9UB63cbpOp8OGDdMsLMz075lh\nYqJBp9MxqKURabfbtNvtgR9XaUcdEeuBfwf+IzM/cZg1Y7mjHlfuqKX6Da1H3fdZ4J7DhbRWn6mp\nKebmdjM5uZVNm7YwObmVubndhrRUoCpTH2cAXwfuArL/8ZHM/Mqide6oVyGnPqT6eK4PSSrcsFsf\nkqSaGNSSVDiDWpIKZ1BLUuEMakkqnEEtSYUzqCWpcAa1JBXOoJakwhnUklQ4g1qSCmdQS1LhDGpJ\nKpxBLUmFM6glqXAGtSQVzqCWpMIZ1JJUuGWDOiLmIuLRiLhzJQqSJD1blR31ZcBbRl2IJGlpywZ1\nZn4D+PkK1CINXbfbZd++fXS73bpLkZ43e9QaW63WXhqNzWzbdhGNxmZarb11lyQ9L5GZyy+KaAD/\nlpkzR1iTVZ5LWgndbpdGYzMLCzcDM8CdTE5uZX7+PqampuouTwIgIsjMWG7d+mF+09nZ2advN5tN\nms3mMJ9eqqzT6bBhwzQLCwf3FjNMTDTodDoGtWrTbrdpt9sDP67qjnqa3o761UdY445axXBHrdWg\n6o66ynjeHuCbwKkR8VBEvHcYBUqjNDU1xdzcbiYnt7Jp0xYmJ7cyN7fbkNaqVGlHXemJ3FGrQN1u\nl06nw/T0tCGt4lTdURvUklSTobU+JEn1MqglqXAGtSQVzqCWpMIZ1JJUOINakgpnUEtS4QxqSSqc\nQS1JhTOoJalwBrUkFc6glqTCGdSSVDiDWpIKZ1BLUuEMakkqnEEtSYUzqCWpcJWCOiLeGhH3RcQP\nI+JDoy5KkvSMKlchXwf8E/AW4LeBnRGxedSFlaTdbtddwkj5+lY3X9/4q7Kjfi3w35k5n5lPAl8A\nzh1tWWUZ978ovr7Vzdc3/qoE9UuBhw/5/Mf9+yRJK8CDiZJUuMjMIy+IeB0wm5lv7X/+YSAz828X\nrTvyE0mSniMzY7k1VYL6BcB/AWcCPwVuBXZm5r3DKFKSdGTrl1uQmb+OiD8FbqDXKpkzpCVp5Sy7\no5Yk1euoDyaO85thImIuIh6NiDvrrmUUIuLkiLgpIn4QEXdFxMV11zRMEXFMRHwnIm7vv75L6q5p\n2CJiXUR8LyKurbuWYYuITkR8v//nd2vd9QxbRBwfEVdExL39f4OnH3bt0eyo+2+G+SG9/vUjwD5g\nR2be97yftCAR8Qbgl8DnM3Om7nqGLSJOBE7MzDsi4jjgNuDccfnzA4iIjZn5RP9Yyy3AxZk5Nv/o\nI+KDwGnApsw8p+56hikiHgBOy8yf113LKETEvwD/mZmXRcR6YGNm/mKptUe7ox7rN8Nk5jeAsfxL\nApCZP8vMO/q3fwncy5jNyGfmE/2bx9A7JjM2vb6IOBk4C/hM3bWMSDCmI8QRsQn4/cy8DCAzDxwu\npOHofxN8M8yYiIhp4PeA79RbyXD1WwO3Az8DvpqZ++quaYj+HvgLxug/n0US+GpE7IuIC+suZshe\nDjwWEZf1W1efjojJwy0ey/+tNJh+2+NK4AP9nfXYyMynMvM1wMnA6RHxqrprGoaIOBt4tP8TUfQ/\nxs0ZmbmF3k8N7++3IsfFemAL8Mn+a3wC+PDhFh9tUP8EeNkhn5/cv0+rRL83diXwr5l5Td31jEr/\nx8qbgbfWXcuQnAGc0+/jtoCtEfH5mmsaqsz8af/XLnA1vVbruPgx8HBmfrf/+ZX0gntJRxvU+4BX\nRkQjIjYAO4BxO/o8rruVgz4L3JOZn6i7kGGLiBdHxPH925PANmAsDpRm5kcy82WZ+Qp6/+5uysx3\n113XsETExv5PekTEscCbgbvrrWp4MvNR4OGIOLV/15nAPYdbv+wbXpb5ZmP9ZpiI2AM0gd+KiIeA\nSw42/8dBRJwB/BFwV7+Pm8BHMvMr9VY2NC8BPtefTloH7M3M62quSdWcAFzdPzXFeuDyzLyh5pqG\n7WLg8oiYAB4A3nu4hb7hRZIK58FESSqcQS1JhTOoJalwBrUkFc6glqTCGdSSVDiDWpIKZ1BLUuH+\nH1pY4+tXGp0xAAAAAElFTkSuQmCC\n","text/plain":"<matplotlib.figure.Figure at 0x1143bf2b0>"},"metadata":{},"output_type":"display_data"}]},{"metadata":{},"cell_type":"markdown","source":"It's clear that we need a more sophisticated model to describe the relationship between $x$ and $y$.\n\nOne approach to this is to transform the data, adding extra columns of features to drive more flexibility in the model.\nFor example, we can add polynomial features to the data this way:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import PolynomialFeatures\npoly = PolynomialFeatures(degree=3, include_bias=False)\nX2 = poly.fit_transform(X)\nprint(X2)","execution_count":12,"outputs":[{"name":"stdout","output_type":"stream","text":"[[   1.    1.    1.]\n [   2.    4.    8.]\n [   3.    9.   27.]\n [   4.   16.   64.]\n [   5.   25.  125.]]\n"}]},{"metadata":{},"cell_type":"markdown","source":"The derived feature matrix has one column representing $x$, and a second column representing $x^2$, and a third column representing $x^3$.\nComputing a linear regression on this expanded input gives a much closer fit to our data:"},{"metadata":{"trusted":false},"cell_type":"code","source":"model = LinearRegression().fit(X2, y)\nyfit = model.predict(X2)\nplt.scatter(x, y)\nplt.plot(x, yfit);","execution_count":13,"outputs":[{"data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAWoAAAEACAYAAACatzzfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGaZJREFUeJzt3X1wleWd//H3F8NDkKdWUqhlTECF1K0ZpT7UH2pPBgUV\nf2V1rLJiV1yndrfr4qpFKp0OqaOCTlXYLtpVaWR/gxGlgq1FRQaPVNsGVlRUoK3gAUSxp91SLEYe\nr98f14FiJOSc5D657nOfz2smw0lyJ3yOJl++57qvB3POISIi8dUtdAARETkyFWoRkZhToRYRiTkV\nahGRmFOhFhGJORVqEZGYy6tQm9mNZvamma0xs/lm1qPYwURExGu3UJvZscC/ASOdc3VABTCh2MFE\nRMSryPO6o4CjzWw/0Bt4r3iRRETkUO121M6594B7gM3AVmC7c25ZsYOJiIiXz9DHAGA8UA0cC/Qx\nsyuLHUxERLx8hj7OAzY65/4XwMyeBP4P8OihF5mZNg0RESmQc87auyafWR+bga+YWS8zM2A0sK6N\nvzCRb9OnTw+eQc9Pz0/PL3lv+cpnjHolsBB4FXgdMODBvP8GERHplLxmfTjnfgD8oMhZRETkMLQy\nMQ+pVCp0hKLS8ytten7JZ4WMkxzxG5m5qL6XiEg5MDNcRDcTRUQkIBVqEZGYU6EWEYk5FWoRkZhT\noRYRiTkVahGRmFOhFhGJORVqEZGYU6EWEYk5FWoRkZhToRYRiTkVahGRmFOhFhGJORVqEZGYU6EW\nEYk5FWoRkZhToRYRibl2C7WZDTezV81sde7Pv5jZ5K4IJyIiBR7FZWbdgHeBM51zW1p9TkdxiYgU\nIN+juPI6hfwQ5wEbWhdpEZFiWLHif+nVawNDh9ZQVVUVOk4whY5RXwE0FSOIiMih5sz5GV/9qnHe\neVOprq6lqWlB6EjB5D30YWbdgfeAk5xz2cN8XkMfIhKJbDbLscc+zt69lwNVwBoqK+vZtGl9ojrr\nYgx9XAi8crgifUBDQ8PBx6lUilQqVcC3FxHxXnttC/v2XQX0z32kju7dq8lkMiVdqNPpNOl0uuCv\nK6SjbgKedc7Na+Pz6qhFJBJTpuzkvvsWsW9fHVBHuXfUeY1Rm1lv/I3EJzsbTETkSD78EB555Gju\nvvszVFbW06/fSCor65k79/5EFelCFDQ974jfSB21iETgnnuguRkef9yPVWcyGWpqkjnrI9+OWoVa\nRGJj1y4YNgyefhpOPTV0muKLdOhDRKQrzJsHdXXlUaQLoY5aRGJh716orYXGRjjnnNBpuoY6ahEp\nKU88AYMHl0+RLkShS8hFRCLnHMyYATNnhk4ST+qoRSS4X/wCunWDCy8MnSSeVKhFJCjn4M474dZb\nwdodrS1PKtQiEtSKFZDNwmWXhU4SXyrUIhLUjBkwdSocdVToJPGlm4kiEswrr8Cbb8JTT4VOEm/q\nqEUkmJkz4eaboWfP0EniTQteRCSI9evh3HNh40bo0yd0mjC04EVEYu3uu+H668u3SBdCHbWIdLnN\nm+GUU+Dtt+Gznw2dJhx11CISW/fcA9deW95FuhDqqEWkS2WzMHw4vPUWHHts6DRhqaMWkViaPRuu\nuEJFuhDqqEWky+zY4Q8GaG6G448PnSY8ddQiEjsPPABjx6pIFyqvjtrM+gMPA18C9gP/5JxrbnWN\nOmoRaVNLCwwdCs8/DyefHDpNPOTbUee7hHw2sMQ593UzqwB6dyqdiJSdxkY44wwV6Y5ot6M2s37A\nq865I75YUUctIm3ZswdOPBGamuCss0KniY8ox6iHAn80s0YzW21mD5pZZecjiki5eOwxP+yhIt0x\n+Qx9VAAjgX91zv2Pmc0CvgtMb31hQ0PDwcepVIpUKhVNShEpWfv3+61MZ80KnSS8dDpNOp0u+Ovy\nGfoYBPzaOTcs9/7ZwFTn3P9tdZ2GPkTkUxYvhttvh1WrdIJLa5ENfTjnPgC2mNnw3IdGA2s7mU9E\nyoCO2YpGvrM+JgPzzaw7sBG4pniRRCQpli+HDz+ESy4JnaS0aWWiiBTN6NHwjW/ApEmhk8STViaK\nSFDNzX4b04kTQycpfSrUIlIUM2bAd74D3buHTlL6NPQhIpF76y0/7LFxI/TWOuY2aehDRIKZORMm\nT1aRjoo6ahGJ1DvvwGmnwYYNMGBA6DTxpo5aRIL44Q/huutUpKOkjlpEIrNtG5x0EqxbB4MGhU4T\nf+qoRaTLzZoFV16pIh01ddQiEont2/3JLatXQ3V16DSlQR21iHSpOXPg4otVpItBHbWIdNpHH/n9\nptNp+OIXQ6cpHeqoRaTLPPwwjBqlIl0s6qhFpFN274YTToCf/hROPz10mtKijlpEusT8+TBihIp0\nMamjFpEO27fPz5v+8Y+hvj50mtKjjlpEim7RIvjMZ0DHoxaXCrWIdMiBY7amTdMxW8WmQi0iHbJ0\nqb+RePHFoZMknwq1iHTIgUNru6mKFF1eh9uaWQb4C7Af2OOcO6OYoUQk3l5+GbZsgSuuCJ2kPOR7\nCvl+IOWc+3Mxw4hIaZgxA265BSryrSDSKfn+ZzY0TCIiwOuv+42XFi4MnaR85Ft8HfC8ma0ys28W\nM5CIxNvMmXDjjdCrV+gk5SPfjnqUc+59M6vCF+x1zrmXWl/U0NBw8HEqlSKlyZUiifL22/D88/Dg\ng6GTlKZ0Ok06nS746wpemWhm04EPnXP3tvq4ViaKJNx118HgwXDbbaGTJEO+KxPb7ajNrDfQzTn3\nVzM7GhgD/CCCjCJSQrZu9ePSv/td6CTlJ5+hj0HAIjNzuevnO+eWFjeWiMTNvffC1VfDwIGhk5Qf\nbcokIu3605/gxBNhzRoYMiR0muTQpkwiEpkf/QguvVRFOhR11CJyRB9+CMOG+dWIw4eHTpMs6qhF\nJBIPPuj3mlaRDkcdtYi0adcu300//TScemroNMmjjlpEOm3ePKirU5EOTR21iBzW3r1QWwuNjXDO\nOaHTJJM6ahHplIUL/SpEFenwtEmhiHyKc34r0xkzQicRUEctIoexZIk/B/HCC0MnEVChFpFWDhxa\ne+utOrQ2LlSoReQTfvlL+MMf4LLLQieRA1SoReQT7rwTpk6Fo44KnUQO0M1EETlo9Wp480146qnQ\nSeRQ6qhF5KAZM+Dmm6Fnz9BJ5FBa8CIiAPz2t37O9MaN0KdP6DTlQQteRKQgd90F11+vIh1H6qhF\nhM2b4ZRT/OG1n/1s6DTlQx21iOTtnnvg2mtVpONKHbVImctmYcQIP9vj2GNDpykvkXfUZtbNzFab\n2c86F01E4mT2bLj8chXpOCtkHvUNwFqgX5GyxFI2myWTyVBTU0NVVVXoOCKR2rEDfvxjaG4OnUSO\nJK+O2syGABcBDxc3Trw0NS3guOPO4NxzX+S442ppaloQOpJIpB54AMaOheOPD51EjiSvMWozewK4\nA+gP3Oyc+9phrknUGHU2m6W6upaWlheBLwHbqKz8OzZtWq/OWhKhpcUfs7V0KZx8cug05SnfMep2\nhz7MbBzwgXPuNTNLAW1+04aGhoOPU6kUqVQqn6yxlMlk6NGjhpaWL+U+MhjnbiKTyahQSyI0NsLp\np6tId6V0Ok06nS7469rtqM3sTuAqYC9QCfQFnnTO/WOr6xLaUb8A1AHrgQE0NlYyaVL/wOlEOmfP\nHn+q+KOPwllnhU5TviKb9eGcm+acO845NwyYACxvXaSTqKqqirlz76eysp5+/UZSWTmK229fwy23\n9GfFitDpRDrnscegpkZFulQUNI/azL5KmYxRH9B61seyZTBxIixbppeMUpr27/c/u/fdB2PGhE5T\n3vLtqLXgpQMeewymTIGXXoLq6tBpRAqzeDHcfjusWqUTXEKL7GaifNqECbBtG1xwgT8NY+DA0IlE\n8nPg0Fods1VatNdHB/37v8P48XDxxbBzZ+g0IvlZvtwvcrnkktBJpBAa+ugE5+Caa/xeCYsXQ/fu\noROJHNl558FVV8GkSaGTCGj3vC5hBg895B9/85u+cIvE1cqV8Pvf+5vhUlpUqDupe3d4/HFYvx6m\nTQudRqRtM2bAd76jV36lSEMfEfnjH+Hss+Ff/gVuuCF0GpFPeustGD3aH7PVu3foNHKAZn10sYED\n4bnnfLEeNMjPDBGJi7vugsmTVaRLlTrqiL3xhu9cHn3U37gRCe2dd+C002DDBhgwIHQaOZRuJgZy\n8smwcCFceSWsXh06jQj88Idw3XUq0qVMHXWRPPmkP9H5l7/UXr8SzrZtcNJJsG6dH5KTeNEYdWCX\nXurnV48dCy+/rF8SCWPWLP/qTj9/pU0ddZE1NMDPfw7pNPTtGzqNlJPt2/2rudWrtSdNXGmMOiam\nT/c3ci69FHbvDp1GysmcOX6LAxXp0qeOugvs2weXXQa9esH8+dBN/zxKkX30EQwd6l/JffGLodNI\nW9RRx8hRR/npelu3wk03aam5FN/DD8OoUSrSSaGOugtt3w7nnOM3xZk6NXQaSardu+GEE+CnP/Vn\nIkp8adZHDA0YAM8+6zudwYPh6qtDJ5Ikmj8fRoxQkU4SFeou9oUv+GKdSvll5+PGhU4kSbJvn18u\n/sADoZNIlDRGHUBtrd+/etIk+M1vQqeRJFm0yL9yS6VCJ5EotVuozaynmTWb2atm9oaZTe+KYEn3\nla/AI4/A3/+93yJVpLMOHLM1bZqO2Uqadgu1c24XUO+cOxU4BbjQzM4oerIyMG4czJzpz17cujV0\nGilF2WyWVatWkc1mWboUdu3yc6clWfIao3bOfZR72DP3NZreEZFJkz55UK42zpF8NTUt4Nprv02P\nHjXs3p2hpmYd3/ve5zRPP4Hymp5nZt2AV4DjgTnOuVsPc42m53WQc3DjjX6p73PPQWVl6EQSd9ls\nlurqWlpaXgDqgLcxq2Dr1qP5/OerQseTPEU6Pc85tx841cz6AYvN7CTn3NrW1zU0NBx8nEqlSOmO\nRl7M4N57/Vl2EyfCE0/4RTIibclkMvToUUNLS13uIyfQs+cdvPvuGBXqGEun06TT6YK/ruAFL2b2\nfWCnc+7eVh9XR91Ju3b5cesTTvDTq3RDSNry6Y56D716VbN58+tUValQl4rIlpCb2UAz6597XAmc\nD2ieQhH07On3sV65Em67LXQaibOqqirmzr2fysp6KiqeoaLidn7yk/tUpBOq3Y7azE4G5uGLejdg\ngXPujsNcp446Ih984FcvTpkC3/pW6DQSZ83Nf2Ls2H68+upfGDp0YOg4UqB8O2rt9RFTGzb4fUHm\nzIFLLgmdRuJo715/T2PECL0CK1Uq1AmwerWftrdwIZx7bug0Eid/+IM/6b6iwv989OsXOpF0hLY5\nTYCRI/32qF//uj/dXAT8PYzTToOzzoJnnlGRLgcq1DF33nkwezZcdBFs2hQ6jYT20EN+5eGPfgR3\n3KFpnOVCu+eVgAkT/A3GA6sXB+qeUdn5+GN/qv2vfw0vvQTDh4dOJF1JHXWJuOEGGD/ed1M7d4ZO\nI11p0yZ/Y3nHDmhuVpEuRyrUJWTGDL9F6uWXw549odNIV1i2DM4807+qWrAA+vQJnUhC0KyPErNn\nj98ataoKGhu1ejGpnPMHAPzHf/gbytqNIZk0PS/Bdu6E0aOhvt532ZIsO3b4XRXfe89PvRsyJHQi\nKRZNz0uwo4+Gp5/2p3nMnh06jURp7Vo44wwYNAhefFFFWjzN+ihRAwf6LVHPPtv/Uk+YEDqRdNbC\nhfDtb8Pdd/uOWuQAFeoSVl0NS5b4YZCBA/2cayk9e/f647OeeMIvYPnyl0MnkrjRGHUCrFgBl13m\nTzcfOTJ0GinEoUvBm5rgmGNCJ5KupDHqMnLuufBf/+XnWG/YEDqN5Kv1UnAVaWmLhj4S4pJLfHc2\ndiy8/LIft5b4eugh+N73/J/jx4dOI3GnQp0g3/oWvP++3xcknYa+fUMnkta0FFw6QkMfCTN9Opx+\nOlx6KezeHTqNHEpLwaWjVKgTxswfNtC3L1x9NezfHzqRgJaCS+do1kdCffwxjBnjZ4Hcd5+Wmoei\npeByJFpCLmzf7l9qX3UVTJ0aOk350VJwaU+Up5APMbPlZvaWmb1hZpOjiSjFNmCAn1v9wAMwb17o\nNOVFS8ElSvmcQj4YGOyce83M+gCvAOOdc+tbXaeOOqbWr/cvuefOhXHjPvm5bDZLJpOhpqaGqqqq\nIPmSRkvBJV+RddTOuW3Ouddyj/8KrAO+0PmI0lVqa2HxYl80fvObv328qWkB1dW1nH/+P1NdXUtT\n04JgGZNg71645RaYMsUvYFGRlqgUNEZtZjVAGvhSrmgf+jl11DH3i1/Atdf6OdbHHJOlurqWlpYX\ngDpgDZWV9WzatF6ddQdoKbh0RL4ddd4LXnLDHguBG1oX6QMaGhoOPk6lUqR0iztWxo2DmTP92Ytz\n5mylR48aWlrqcp+to3v3ajKZjAp1gVau9HutfOMbcNttOnBW2pZOp0mn0wV/XV4dtZlVAE8Dzzjn\nDrsDsjrq0nHXXTBv3l7eeedEPv74KdRRd5yWgktnRN1R/wRY21aRltJyyy3w/vsVPPdcM5nMqfTo\nMYg9ezYxd+79KtJ50lJw6Ur5zPoYBawA3gBc7m2ac+7ZVtepoy4h+/fDxImwY8cuvv/9NRx/vGZ9\n5GvTJj/UMWyYn0mjVYbSUVrwIu3atcuPW1dU+KI9dix87nOhU8XbsmV+AdGUKXDTTVrxKZ2jQi15\n2bnTL21+5hlYvhxOPNHvvnfhhX5zJ90Y87QUXIpBhVoKtns3/OpXvmgvWeK3TB0zxhftcu62tRRc\nikWFWjptyxa/BL2cu+21a/2WsfX1MGsW9OwZOpEkiQq1RGr3bn9yzDPP+Ldy6La1FFyKTYVaiqp1\ntz18uC/aSei2Dz0VfOFCnQouxaNCLV0mSd22loJLV1KhlmBKtdvWUnDpairUEgul0m1rKbiEoEIt\nsRS3bvvQpeCLFmkpuHQtFWqJvdDdtpaCS2gq1FJyDnTbS5bACy8Ut9vWUnCJAxVqKWltddsXXeS7\n7Y7uH6Wl4BInKtSSKFF021oKLnGjQi2Jdbhue+zYv41tH67b1lJwiSMVaikbW7b8rWgvXw4jRnyy\n2160SEvBJZ5UqKUste62t26F/v21FFziSYVaBHj3XV+o+/YNnUTk01SoRURiLt9C3S2PbzTXzD4w\nszXRRBMRkUK0W6iBRmBssYOIiMjhtVuonXMvAX/ugiwikctms6xatYpsNhs6ikiH5dNRi5SkpqYF\nVFfXcv75/0x1dS1NTQtCRxLpkLxuJppZNfBz51zdEa7RzUSJjWw2S3V1LS0tLwB1wBoqK+vZtGk9\nVR1dfy4SsXxvJlZE+Zc2NDQcfJxKpUhpIwUJJJPJ0KNHDS0tB3qLOrp3ryaTyahQSzDpdJp0Ol3w\n1+XbUdfgO+qTj3CNOmqJDXXUUgqinJ73KPArYLiZbTaza6IIKFJMVVVVzJ17P5WV9fTrN5LKynrm\nzr1fRVpKkha8SKJls1kymQw1NTUq0hI7WpkoIhJzkQ19iIhIWCrUIiIxp0ItIhJzKtQiIjGnQi0i\nEnMq1CIiMadCLSIScyrUIiIxp0ItIhJzKtQiIjGnQi0iEnMq1CIiMadCLSIScyrUIiIxp0ItIhJz\nKtQiIjGnQi0iEnMq1CIiMZdXoTazC8xsvZn9zsymFjuUiIj8TT6nkHcD/hMYC/wd8A9mVlvsYHGS\nTqdDRygqPb/SpueXfPl01GcAv3fObXLO7QEeA8YXN1a8JP0HRc+vtOn5JV8+hfoLwJZD3n839zER\nEekCupkoIhJz5pw78gVmXwEanHMX5N7/LuCcc3e1uu7I30hERD7FOWftXZNPoT4K+C0wGngfWAn8\ng3NuXRQhRUTkyCrau8A5t8/MrgeW4odK5qpIi4h0nXY7ahERCavTNxOTvBjGzOaa2QdmtiZ0lmIw\nsyFmttzM3jKzN8xscuhMUTKznmbWbGav5p7f9NCZomZm3cxstZn9LHSWqJlZxsxez/3/Wxk6T9TM\nrL+ZPWFm63K/g2e2eW1nOurcYpjf4cev3wNWAROcc+s7/E1jxMzOBv4K/Ldzri50nqiZ2WBgsHPu\nNTPrA7wCjE/K/z8AM+vtnPsod6/lZWCycy4xv/RmdiPwZaCfc+5rofNEycw2Al92zv05dJZiMLNH\ngBedc41mVgH0ds7tONy1ne2oE70Yxjn3EpDIHxIA59w259xrucd/BdaRsDnyzrmPcg974u/JJGas\nz8yGABcBD4fOUiRGQqcQm1k/4BznXCOAc25vW0UaOv8fQYthEsLMaoBTgOawSaKVGxp4FdgGPO+c\nWxU6U4TuA6aQoH98WnHA82a2ysy+GTpMxIYCfzSzxtzQ1YNmVtnWxYn810oKkxv2WAjckOusE8M5\nt985dyowBDjTzE4KnSkKZjYO+CD3ishyb0kzyjk3Ev+q4V9zQ5FJUQGMBObknuNHwHfburizhXor\ncNwh7w/JfUxKRG5sbCHw/5xzT4XOUyy5l5UvABeEzhKRUcDXcuO4TUC9mf134EyRcs69n/szCyzC\nD7UmxbvAFufc/+TeX4gv3IfV2UK9CjjBzKrNrAcwAUja3eekdisH/ARY65ybHTpI1MxsoJn1zz2u\nBM4HEnGj1Dk3zTl3nHNuGP73brlz7h9D54qKmfXOvdLDzI4GxgBvhk0VHefcB8AWMxue+9BoYG1b\n17e74KWdvyzRi2HM7FEgBRxjZpuB6QcG/5PAzEYBE4E3cuO4DpjmnHs2bLLIfB6Yl5ud1A1Y4Jxb\nEjiT5GcQsCi3NUUFMN85tzRwpqhNBuabWXdgI3BNWxdqwYuISMzpZqKISMypUIuIxJwKtYhIzKlQ\ni4jEnAq1iEjMqVCLiMScCrWISMypUIuIxNz/B0W/Syq+z56RAAAAAElFTkSuQmCC\n","text/plain":"<matplotlib.figure.Figure at 0x114819c50>"},"metadata":{},"output_type":"display_data"}]},{"metadata":{},"cell_type":"markdown","source":"This idea of improving a model not by changing the model, but by transforming the inputs, is fundamental to many of the more powerful machine learning methods.\nWe explore this idea further in [In Depth: Linear Regression](05.06-Linear-Regression.ipynb) in the context of *basis function regression*.\nMore generally, this is one motivational path to the powerful set of techniques known as *kernel methods*, which we will explore in [In-Depth: Support Vector Machines](05.07-Support-Vector-Machines.ipynb)."},{"metadata":{},"cell_type":"markdown","source":"## Imputation of Missing Data\n\nAnother common need in feature engineering is handling of missing data.\nWe discussed the handling of missing data in ``DataFrame``s in [Handling Missing Data](03.04-Missing-Values.ipynb), and saw that often the ``NaN`` value is used to mark missing values.\nFor example, we might have a dataset that looks like this:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from numpy import nan\nX = np.array([[ nan, 0,   3  ],\n              [ 3,   7,   9  ],\n              [ 3,   5,   2  ],\n              [ 4,   nan, 6  ],\n              [ 8,   8,   1  ]])\ny = np.array([14, 16, -1,  8, -5])","execution_count":14,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When applying a typical machine learning model to such data, we will need to first replace such missing data with some appropriate fill value.\nThis is known as *imputation* of missing values, and strategies range from simple (e.g., replacing missing values with the mean of the column) to sophisticated (e.g., using matrix completion or a robust model to handle such data).\n\nThe sophisticated approaches tend to be very application-specific, and we won't dive into them here.\nFor a baseline imputation approach, using the mean, median, or most frequent value, Scikit-Learn provides the ``Imputer`` class:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.preprocessing import Imputer\nimp = Imputer(strategy='mean')\nX2 = imp.fit_transform(X)\nX2","execution_count":15,"outputs":[{"data":{"text/plain":"array([[ 4.5,  0. ,  3. ],\n       [ 3. ,  7. ,  9. ],\n       [ 3. ,  5. ,  2. ],\n       [ 4. ,  5. ,  6. ],\n       [ 8. ,  8. ,  1. ]])"},"execution_count":15,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"We see that in the resulting data, the two missing values have been replaced with the mean of the remaining values in the column. This imputed data can then be fed directly into, for example, a ``LinearRegression`` estimator:"},{"metadata":{"trusted":false},"cell_type":"code","source":"model = LinearRegression().fit(X2, y)\nmodel.predict(X2)","execution_count":16,"outputs":[{"data":{"text/plain":"array([ 13.14869292,  14.3784627 ,  -1.15539732,  10.96606197,  -5.33782027])"},"execution_count":16,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"## Feature Pipelines\n\nWith any of the preceding examples, it can quickly become tedious to do the transformations by hand, especially if you wish to string together multiple steps.\nFor example, we might want a processing pipeline that looks something like this:\n\n1. Impute missing values using the mean\n2. Transform features to quadratic\n3. Fit a linear regression\n\nTo streamline this type of processing pipeline, Scikit-Learn provides a ``Pipeline`` object, which can be used as follows:"},{"metadata":{"trusted":false},"cell_type":"code","source":"from sklearn.pipeline import make_pipeline\n\nmodel = make_pipeline(Imputer(strategy='mean'),\n                      PolynomialFeatures(degree=2),\n                      LinearRegression())","execution_count":17,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This pipeline looks and acts like a standard Scikit-Learn object, and will apply all the specified steps to any input data."},{"metadata":{"trusted":false},"cell_type":"code","source":"model.fit(X, y)  # X with missing values, from above\nprint(y)\nprint(model.predict(X))","execution_count":18,"outputs":[{"name":"stdout","output_type":"stream","text":"[14 16 -1  8 -5]\n[ 14.  16.  -1.   8.  -5.]\n"}]}],"metadata":{"anaconda-cloud":{},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}