python-datascience/03.06-Concat-And-Append.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Combining Datasets: Concat and Append"},{"metadata":{},"cell_type":"markdown","source":"Some of the most interesting studies of data come from combining different data sources.\nThese operations can involve anything from very straightforward concatenation of two different datasets, to more complicated database-style joins and merges that correctly handle any overlaps between the datasets.\n``Series`` and ``DataFrame``s are built with this type of operation in mind, and Pandas includes functions and methods that make this sort of data wrangling fast and straightforward.\n\nHere we'll take a look at simple concatenation of ``Series`` and ``DataFrame``s with the ``pd.concat`` function; later we'll dive into more sophisticated in-memory merges and joins implemented in Pandas.\n\nWe begin with the standard imports:"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"import pandas as pd\nimport numpy as np","execution_count":1,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For convenience, we'll define this function which creates a ``DataFrame`` of a particular form that will be useful below:"},{"metadata":{"trusted":false},"cell_type":"code","source":"def make_df(cols, ind):\n    \"\"\"Quickly make a DataFrame\"\"\"\n    data = {c: [str(c) + str(i) for i in ind]\n            for c in cols}\n    return pd.DataFrame(data, ind)\n\n# example DataFrame\nmake_df('ABC', range(3))","execution_count":2,"outputs":[{"data":{"text/html":"<div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n      <td>C0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n  </tbody>\n</table>\n</div>","text/plain":"    A   B   C\n0  A0  B0  C0\n1  A1  B1  C1\n2  A2  B2  C2"},"execution_count":2,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"In addition, we'll create a quick class that allows us to display multiple ``DataFrame``s side by side. The code makes use of the special ``_repr_html_`` method, which IPython uses to implement its rich object display:"},{"metadata":{"collapsed":true,"trusted":false},"cell_type":"code","source":"class display(object):\n    \"\"\"Display HTML representation of multiple objects\"\"\"\n    template = \"\"\"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>{0}</p>{1}\n    </div>\"\"\"\n    def __init__(self, *args):\n        self.args = args\n        \n    def _repr_html_(self):\n        return '\\n'.join(self.template.format(a, eval(a)._repr_html_())\n                         for a in self.args)\n    \n    def __repr__(self):\n        return '\\n\\n'.join(a + '\\n' + repr(eval(a))\n                           for a in self.args)\n    ","execution_count":3,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The use of this will become clearer as we continue our discussion in the following section."},{"metadata":{},"cell_type":"markdown","source":"## Recall: Concatenation of NumPy Arrays\n\nConcatenation of ``Series`` and ``DataFrame`` objects is very similar to concatenation of Numpy arrays, which can be done via the ``np.concatenate`` function as discussed in [The Basics of NumPy Arrays](02.02-The-Basics-Of-NumPy-Arrays.ipynb).\nRecall that with it, you can combine the contents of two or more arrays into a single array:"},{"metadata":{"trusted":false},"cell_type":"code","source":"x = [1, 2, 3]\ny = [4, 5, 6]\nz = [7, 8, 9]\nnp.concatenate([x, y, z])","execution_count":4,"outputs":[{"data":{"text/plain":"array([1, 2, 3, 4, 5, 6, 7, 8, 9])"},"execution_count":4,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"The first argument is a list or tuple of arrays to concatenate.\nAdditionally, it takes an ``axis`` keyword that allows you to specify the axis along which the result will be concatenated:"},{"metadata":{"trusted":false},"cell_type":"code","source":"x = [[1, 2],\n     [3, 4]]\nnp.concatenate([x, x], axis=1)","execution_count":5,"outputs":[{"data":{"text/plain":"array([[1, 2, 1, 2],\n       [3, 4, 3, 4]])"},"execution_count":5,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"## Simple Concatenation with ``pd.concat``"},{"metadata":{},"cell_type":"markdown","source":"Pandas has a function, ``pd.concat()``, which has a similar syntax to ``np.concatenate`` but contains a number of options that we'll discuss momentarily:\n\n```python\n# Signature in Pandas v0.18\npd.concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,\n          keys=None, levels=None, names=None, verify_integrity=False,\n          copy=True)\n```\n\n``pd.concat()`` can be used for a simple concatenation of ``Series`` or ``DataFrame`` objects, just as ``np.concatenate()`` can be used for simple concatenations of arrays:"},{"metadata":{"trusted":false},"cell_type":"code","source":"ser1 = pd.Series(['A', 'B', 'C'], index=[1, 2, 3])\nser2 = pd.Series(['D', 'E', 'F'], index=[4, 5, 6])\npd.concat([ser1, ser2])","execution_count":6,"outputs":[{"data":{"text/plain":"1    A\n2    B\n3    C\n4    D\n5    E\n6    F\ndtype: object"},"execution_count":6,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"It also works to concatenate higher-dimensional objects, such as ``DataFrame``s:"},{"metadata":{"trusted":false},"cell_type":"code","source":"df1 = make_df('AB', [1, 2])\ndf2 = make_df('AB', [3, 4])\ndisplay('df1', 'df2', 'pd.concat([df1, df2])')","execution_count":7,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df2</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>A4</td>\n      <td>B4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df1, df2])</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>A4</td>\n      <td>B4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df1\n    A   B\n1  A1  B1\n2  A2  B2\n\ndf2\n    A   B\n3  A3  B3\n4  A4  B4\n\npd.concat([df1, df2])\n    A   B\n1  A1  B1\n2  A2  B2\n3  A3  B3\n4  A4  B4"},"execution_count":7,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"By default, the concatenation takes place row-wise within the ``DataFrame`` (i.e., ``axis=0``).\nLike ``np.concatenate``, ``pd.concat`` allows specification of an axis along which concatenation will take place.\nConsider the following example:"},{"metadata":{"trusted":false},"cell_type":"code","source":"df3 = make_df('AB', [0, 1])\ndf4 = make_df('CD', [0, 1])\ndisplay('df3', 'df4', \"pd.concat([df3, df4], axis='col')\")","execution_count":8,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df3</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df4</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>C0</td>\n      <td>D0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>C1</td>\n      <td>D1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df3, df4], axis='col')</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n      <td>C0</td>\n      <td>D0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n      <td>D1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df3\n    A   B\n0  A0  B0\n1  A1  B1\n\ndf4\n    C   D\n0  C0  D0\n1  C1  D1\n\npd.concat([df3, df4], axis='col')\n    A   B   C   D\n0  A0  B0  C0  D0\n1  A1  B1  C1  D1"},"execution_count":8,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"We could have equivalently specified ``axis=1``; here we've used the more intuitive ``axis='col'``. "},{"metadata":{},"cell_type":"markdown","source":"### Duplicate indices\n\nOne important difference between ``np.concatenate`` and ``pd.concat`` is that Pandas concatenation *preserves indices*, even if the result will have duplicate indices!\nConsider this simple example:"},{"metadata":{"trusted":false},"cell_type":"code","source":"x = make_df('AB', [0, 1])\ny = make_df('AB', [2, 3])\ny.index = x.index  # make duplicate indices!\ndisplay('x', 'y', 'pd.concat([x, y])')","execution_count":9,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>x</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>y</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([x, y])</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>0</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"x\n    A   B\n0  A0  B0\n1  A1  B1\n\ny\n    A   B\n0  A2  B2\n1  A3  B3\n\npd.concat([x, y])\n    A   B\n0  A0  B0\n1  A1  B1\n0  A2  B2\n1  A3  B3"},"execution_count":9,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"Notice the repeated indices in the result.\nWhile this is valid within ``DataFrame``s, the outcome is often undesirable.\n``pd.concat()`` gives us a few ways to handle it."},{"metadata":{},"cell_type":"markdown","source":"#### Catching the repeats as an error\n\nIf you'd like to simply verify that the indices in the result of ``pd.concat()`` do not overlap, you can specify the ``verify_integrity`` flag.\nWith this set to True, the concatenation will raise an exception if there are duplicate indices.\nHere is an example, where for clarity we'll catch and print the error message:"},{"metadata":{"trusted":false},"cell_type":"code","source":"try:\n    pd.concat([x, y], verify_integrity=True)\nexcept ValueError as e:\n    print(\"ValueError:\", e)","execution_count":10,"outputs":[{"name":"stdout","output_type":"stream","text":"ValueError: Indexes have overlapping values: [0, 1]\n"}]},{"metadata":{},"cell_type":"markdown","source":"#### Ignoring the index\n\nSometimes the index itself does not matter, and you would prefer it to simply be ignored.\nThis option can be specified using the ``ignore_index`` flag.\nWith this set to true, the concatenation will create a new integer index for the resulting ``Series``:"},{"metadata":{"trusted":false},"cell_type":"code","source":"display('x', 'y', 'pd.concat([x, y], ignore_index=True)')","execution_count":11,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>x</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>y</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([x, y], ignore_index=True)</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"x\n    A   B\n0  A0  B0\n1  A1  B1\n\ny\n    A   B\n0  A2  B2\n1  A3  B3\n\npd.concat([x, y], ignore_index=True)\n    A   B\n0  A0  B0\n1  A1  B1\n2  A2  B2\n3  A3  B3"},"execution_count":11,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"#### Adding MultiIndex keys\n\nAnother option is to use the ``keys`` option to specify a label for the data sources; the result will be a hierarchically indexed series containing the data:"},{"metadata":{"trusted":false},"cell_type":"code","source":"display('x', 'y', \"pd.concat([x, y], keys=['x', 'y'])\")","execution_count":12,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>x</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>y</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([x, y], keys=['x', 'y'])</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th rowspan=\"2\" valign=\"top\">x</th>\n      <th>0</th>\n      <td>A0</td>\n      <td>B0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th rowspan=\"2\" valign=\"top\">y</th>\n      <th>0</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"x\n    A   B\n0  A0  B0\n1  A1  B1\n\ny\n    A   B\n0  A2  B2\n1  A3  B3\n\npd.concat([x, y], keys=['x', 'y'])\n      A   B\nx 0  A0  B0\n  1  A1  B1\ny 0  A2  B2\n  1  A3  B3"},"execution_count":12,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"The result is a multiply indexed ``DataFrame``, and we can use the tools discussed in [Hierarchical Indexing](03.05-Hierarchical-Indexing.ipynb) to transform this data into the representation we're interested in."},{"metadata":{},"cell_type":"markdown","source":"### Concatenation with joins\n\nIn the simple examples we just looked at, we were mainly concatenating ``DataFrame``s with shared column names.\nIn practice, data from different sources might have different sets of column names, and ``pd.concat`` offers several options in this case.\nConsider the concatenation of the following two ``DataFrame``s, which have some (but not all!) columns in common:"},{"metadata":{"trusted":false},"cell_type":"code","source":"df5 = make_df('ABC', [1, 2])\ndf6 = make_df('BCD', [3, 4])\ndisplay('df5', 'df6', 'pd.concat([df5, df6])')","execution_count":13,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df5</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>B</th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>B3</td>\n      <td>C3</td>\n      <td>D3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>B4</td>\n      <td>C4</td>\n      <td>D4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df5, df6])</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NaN</td>\n      <td>B3</td>\n      <td>C3</td>\n      <td>D3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NaN</td>\n      <td>B4</td>\n      <td>C4</td>\n      <td>D4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df5\n    A   B   C\n1  A1  B1  C1\n2  A2  B2  C2\n\ndf6\n    B   C   D\n3  B3  C3  D3\n4  B4  C4  D4\n\npd.concat([df5, df6])\n     A   B   C    D\n1   A1  B1  C1  NaN\n2   A2  B2  C2  NaN\n3  NaN  B3  C3   D3\n4  NaN  B4  C4   D4"},"execution_count":13,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"By default, the entries for which no data is available are filled with NA values.\nTo change this, we can specify one of several options for the ``join`` and ``join_axes`` parameters of the concatenate function.\nBy default, the join is a union of the input columns (``join='outer'``), but we can change this to an intersection of the columns using ``join='inner'``:"},{"metadata":{"trusted":false},"cell_type":"code","source":"display('df5', 'df6',\n        \"pd.concat([df5, df6], join='inner')\")","execution_count":14,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df5</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>B</th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>B3</td>\n      <td>C3</td>\n      <td>D3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>B4</td>\n      <td>C4</td>\n      <td>D4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df5, df6], join='inner')</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>B3</td>\n      <td>C3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>B4</td>\n      <td>C4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df5\n    A   B   C\n1  A1  B1  C1\n2  A2  B2  C2\n\ndf6\n    B   C   D\n3  B3  C3  D3\n4  B4  C4  D4\n\npd.concat([df5, df6], join='inner')\n    B   C\n1  B1  C1\n2  B2  C2\n3  B3  C3\n4  B4  C4"},"execution_count":14,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"Another option is to directly specify the index of the remaininig colums using the ``join_axes`` argument, which takes a list of index objects.\nHere we'll specify that the returned columns should be the same as those of the first input:"},{"metadata":{"trusted":false},"cell_type":"code","source":"display('df5', 'df6',\n        \"pd.concat([df5, df6], join_axes=[df5.columns])\")","execution_count":15,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df5</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df6</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>B</th>\n      <th>C</th>\n      <th>D</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>B3</td>\n      <td>C3</td>\n      <td>D3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>B4</td>\n      <td>C4</td>\n      <td>D4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>pd.concat([df5, df6], join_axes=[df5.columns])</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n      <th>C</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n      <td>C1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n      <td>C2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NaN</td>\n      <td>B3</td>\n      <td>C3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NaN</td>\n      <td>B4</td>\n      <td>C4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df5\n    A   B   C\n1  A1  B1  C1\n2  A2  B2  C2\n\ndf6\n    B   C   D\n3  B3  C3  D3\n4  B4  C4  D4\n\npd.concat([df5, df6], join_axes=[df5.columns])\n     A   B   C\n1   A1  B1  C1\n2   A2  B2  C2\n3  NaN  B3  C3\n4  NaN  B4  C4"},"execution_count":15,"metadata":{},"output_type":"execute_result"}]},{"metadata":{},"cell_type":"markdown","source":"The combination of options of the ``pd.concat`` function allows a wide range of possible behaviors when joining two datasets; keep these in mind as you use these tools for your own data."},{"metadata":{},"cell_type":"markdown","source":"### The ``append()`` method\n\nBecause direct array concatenation is so common, ``Series`` and ``DataFrame`` objects have an ``append`` method that can accomplish the same thing in fewer keystrokes.\nFor example, rather than calling ``pd.concat([df1, df2])``, you can simply call ``df1.append(df2)``:"},{"metadata":{"trusted":false},"cell_type":"code","source":"display('df1', 'df2', 'df1.append(df2)')","execution_count":16,"outputs":[{"data":{"text/html":"<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df1</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df2</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>3</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>A4</td>\n      <td>B4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>\n<div style=\"float: left; padding: 10px;\">\n    <p style='font-family:\"Courier New\", Courier, monospace'>df1.append(df2)</p><div>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>A</th>\n      <th>B</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>A1</td>\n      <td>B1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>A2</td>\n      <td>B2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>A3</td>\n      <td>B3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>A4</td>\n      <td>B4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n    </div>","text/plain":"df1\n    A   B\n1  A1  B1\n2  A2  B2\n\ndf2\n    A   B\n3  A3  B3\n4  A4  B4\n\ndf1.append(df2)\n    A   B\n1  A1  B1\n2  A2  B2\n3  A3  B3\n4  A4  B4"},"execution_count":16,"metadata":{},"output_type":"execute_result"}]}],"metadata":{"anaconda-cloud":{},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}