Day3 - synchrotron 2015 training advanced - group 2.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:40bf2d7f8a073ebdb28d3d6576da457aa423fe0c4536842bf2c7d2665595de64"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Day3 - synchrotron 2015 training advanced - group 2"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Last week's homework"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      " - Improve the decorator from last week so that it logs arguments (both normal and keyword)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Simple example of decorators - the built in 'property' decorator\n",
      "# converts a normal function into an 'attribute' / property\n",
      "class A:\n",
      "    @property\n",
      "    def temperature(self):\n",
      "        return 37\n",
      "\n",
      "a = A()\n",
      "# print a.temperature()\n",
      "print a.temperature"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "37\n"
       ]
      }
     ],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "A decorator is just a callable that takes a function as an argument and returns a replacement function."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def report(func):\n",
      "    \n",
      "    def wrapper(a,b):\n",
      "        print 'before', a, b\n",
      "        result = func(a,b)\n",
      "        print 'result is', result\n",
      "        return result\n",
      "    \n",
      "    return wrapper\n",
      "\n",
      "@report\n",
      "def add(a,b):\n",
      "    return a+b\n",
      "\n",
      "print add(1,1)\n",
      "print add(2,2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "before 1 1\n",
        "result is 2\n",
        "2\n",
        "before 2 2\n",
        "result is 4\n",
        "4\n"
       ]
      }
     ],
     "prompt_number": 12
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def report(func):\n",
      "    \n",
      "    def wrapper(*args, **kwargs):\n",
      "        # PRE\n",
      "        args_as_nice_string = [str(a) for a in args]\n",
      "        print \"Calling function '%s' with args: %s and kwargs %s\" % \\\n",
      "                (func.__name__,\n",
      "                 \", \".join(args_as_nice_string),\n",
      "                 kwargs)\n",
      "            \n",
      "        # CALL THE ORIGINAL FUNC\n",
      "        result = func(*args, **kwargs)  # func is either 'add' or 'add_lots' or ....\n",
      "\n",
      "        # POST\n",
      "        print \"\\tResult is '%s'\" % result\n",
      "        return result\n",
      "    \n",
      "    return wrapper\n",
      "\n",
      "@report\n",
      "def add(a,b):\n",
      "    return a+b\n",
      "\n",
      "@report\n",
      "def add_lots(*args):\n",
      "    result = 0\n",
      "    for arg in args:\n",
      "        result += arg\n",
      "    return result\n",
      "\n",
      "print add(1,1)\n",
      "print add(2,2)\n",
      "\n",
      "print add_lots(1,1,1,1,1,2,2,2,3)\n",
      "\n",
      "print add(2,b=200)\n",
      "print add(a=2,b=200)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Calling function 'add' with args: 1, 1 and kwargs {}\n",
        "\tResult is '2'\n",
        "2\n",
        "Calling function 'add' with args: 2, 2 and kwargs {}\n",
        "\tResult is '4'\n",
        "4\n",
        "Calling function 'add_lots' with args: 1, 1, 1, 1, 1, 2, 2, 2, 3 and kwargs {}\n",
        "\tResult is '14'\n",
        "14\n",
        "Calling function 'add' with args: 2 and kwargs {'b': 200}\n",
        "\tResult is '202'\n",
        "202\n",
        "Calling function 'add' with args:  and kwargs {'a': 2, 'b': 200}\n",
        "\tResult is '202'\n",
        "202\n"
       ]
      }
     ],
     "prompt_number": 40
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Aside - how join works.  We use this in the above example to print out the parameters\n",
      "\",\".join(['a','b'])\n",
      "\",\".join([str(1),str(2)])\n",
      "\",\".join([str(i) for i in [1,2]])  # list comprehension"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 177,
       "text": [
        "'1,2'"
       ]
      }
     ],
     "prompt_number": 177
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# example of a decorator that replaces the original function\n",
      "# but doesn't bother to call the original function\n",
      "# - this is typically what you DON'T do\n",
      "\n",
      "def smite(func):\n",
      "    def subtract(*args, **kwargs):\n",
      "        return \"Ha Ha I have subverted you %s %s\" % (args, kwargs)\n",
      "    return subtract\n",
      "\n",
      "@smite\n",
      "def add(a, b):\n",
      "    return a+b\n",
      "\n",
      "print add(1,b=2)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Ha Ha I have subverted you (1,) {'b': 2}\n"
       ]
      }
     ],
     "prompt_number": 53
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "http://stackoverflow.com/questions/12151182/python-precondition-postcondition-for-member-function-how"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# precondition decorator\n",
      "\n",
      "import functools\n",
      "\n",
      "def condition(pre_condition=None, post_condition=None):\n",
      "    def decorator(func):\n",
      "        @functools.wraps(func) # presever name, docstring, etc\n",
      "        def wrapper(*args, **kwargs): #NOTE: no self\n",
      "            if pre_condition is not None:\n",
      "               assert pre_condition(*args, **kwargs)\n",
      "            retval = func(*args, **kwargs) # call original function or method\n",
      "            if post_condition is not None:\n",
      "               assert post_condition(retval)\n",
      "            return retval\n",
      "        return wrapper\n",
      "    return decorator\n",
      "\n",
      "def pre_condition(check):\n",
      "    return condition(pre_condition=check)\n",
      "\n",
      "def post_condition(check):\n",
      "    return condition(post_condition=check)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "@pre_condition(lambda arg: arg > 0)\n",
      "def function(arg):\n",
      "    print \"I got run ok\"\n",
      "\n",
      "function(100)\n",
      "# function(-1)  # will raise an exception\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "I got run ok\n",
        "I got run ok\n"
       ]
      }
     ],
     "prompt_number": 59
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "@post_condition(lambda ret: ret > 0)\n",
      "def function():\n",
      "#     return -100   # fails\n",
      "    return 100   # OK\n",
      "\n",
      "function()\n",
      "\n",
      "@post_condition(lambda ret: ret in ['red', 'yellow', 'green'])\n",
      "def traffic_light():\n",
      "    return \"red\"  # ok\n",
      "#     return \"redk\"  # fails\n",
      "\n",
      "traffic_light()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 65,
       "text": [
        "'red'"
       ]
      }
     ],
     "prompt_number": 65
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "More on properties"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Simple example of decorators - the built in 'property' decorator\n",
      "# converts a normal function into an 'attribute' / property\n",
      "\n",
      "class A(object):   # <<<< MUST inherit from object to\n",
      "    @property\n",
      "    def temperature(self):\n",
      "        return 37\n",
      "    @temperature.setter\n",
      "    def temperature(self, value):\n",
      "        print 'got value', value\n",
      "        if value > 80:\n",
      "            print 'warning its getting hot'\n",
      "            \n",
      "a = A()\n",
      "# print a.temperature()\n",
      "print a.temperature\n",
      "\n",
      "a.temperature = 100\n",
      "print a.temperature"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "37\n",
        "got value 100\n",
        "warning its getting hot\n",
        "37\n"
       ]
      }
     ],
     "prompt_number": 72
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "a bit of background..."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In Python you don't use getters or setter functions (properties) just for the fun of it. You first just use attributes and then later, only if needed, eventually migrate to a property *without having to change the code using your classes.*\n",
      "\n",
      "Properties are useful in that they allow us to handle both setting and getting values in a programmatic way but still allow attributes to be accessed as attributes - which is semantically nicer. \n",
      "\n",
      "Another way of looking at it is that: We can turn 'gets' into 'computations' (essentially) and we can turn 'sets' into 'events.'"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "class Dinosaur:\n",
      "    pass\n",
      "\n",
      "d = Dinosaur()\n",
      "\n",
      "d.x = 200\n",
      "print d.x\n",
      "\n",
      "if not hasattr(d, 'x'): \n",
      "    print 'oops'   # can check if attribute exists\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "200\n"
       ]
      }
     ],
     "prompt_number": 76
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Nicer way, is shows that 'x' is an official attribute of this class\n",
      "# not just something 'tacked on'\n",
      "\n",
      "class Dinosaur(object):\n",
      "    def __init__(self):\n",
      "        self.x = 0\n",
      "     \n",
      "# IMAGINE THIS TEST CODE IS YOUR 'CODE BASE' WHICH USES DINOSAUR\n",
      "d = Dinosaur()\n",
      "def my_code_base(d):\n",
      "    print d.x\n",
      "    d.x = 2000   # setting works\n",
      "    print d.x    # getting works\n",
      "my_code_base(d)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0\n",
        "2000\n"
       ]
      }
     ],
     "prompt_number": 79
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Add properties without changing the interface to using Dinosaur.  \n",
      "# Convert attributes into properties when you want to add behaviour to the getting or setting of an attribute.\n",
      "# getters can 'calculate' values.  setters can validate incoming values.  There are other uses too.\n",
      "\n",
      "class Dinosaur(object):\n",
      "    def __init__(self):\n",
      "        self._x = 0\n",
      "        \n",
      "    @property\n",
      "    def x(self):\n",
      "        return self._x\n",
      "    \n",
      "    @x.setter\n",
      "    def x(self, value):\n",
      "        self._x = value\n",
      "\n",
      "d = Dinosaur()\n",
      "my_code_base(d)\n",
      "\n",
      "# Succesfully kept the 'interface' of the dinosaur but added trigger\n",
      "# methods\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0\n",
        "2000\n"
       ]
      }
     ],
     "prompt_number": 82
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Example of 'adding functionality' during set and get\n",
      "\n",
      "class Dinosaur(object):\n",
      "    def __init__(self):\n",
      "        self._x = 0\n",
      "        self.error_margin = 0.1\n",
      "        \n",
      "    @property\n",
      "    def x(self):\n",
      "        return int(self._x + self.error_margin)\n",
      "    @x.setter\n",
      "    def x(self, value):\n",
      "        if type(value) != int:\n",
      "            raise RuntimeError(\"hey mate you can't send me strings!!\" )\n",
      "        self._x = value\n",
      "        \n",
      "    def setx(self, value):   # Achieves the same effect\n",
      "        if type(value) != int:\n",
      "            raise RuntimeError(\"hey mate you can't send me strings!!\" )\n",
      "        self._x = value\n",
      "        "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So there are perhaps two ways of looking at this:\n",
      " - You have a private representation, an attribute that you **do not want to expose** directly because its too dangerous.  There may be limit checking you want to have or calculations.  So it's ok to write some getter and setter functions.  They don't have to be properties - normal functions are ok.  But later you might want to make it 'more elegant' if it feels right, to make them properties so that the *semantics of attribute access are maintained*.\n",
      " - Another case might be that you are already using attributes and want to upgrade some of them to be triggering methods."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Why bother with properties?  More examples:\n",
      "\n",
      "class Mountain(object):\n",
      "    def __init__(self):\n",
      "        self.lakes = ['blue lake', 'ontair', 'salt', 'red sea']\n",
      "        \n",
      "    @property\n",
      "    def count(self):   # another way of doing it\n",
      "        return len(self.lakes)\n",
      "    \n",
      "    def count2(self):  # one way of doing it\n",
      "        return len(self.lakes)\n",
      "        \n",
      "    @count.deleter\n",
      "    def count(self):  # A new idea - a 'deleter' property\n",
      "        #self.lakes = []\n",
      "        self.lakes.pop()\n",
      "        \n",
      "m = Mountain()\n",
      "print m.count\n",
      "del m.count  # triggers the deleter method\n",
      "print m.count\n",
      "del m.count  # triggers the deleter method\n",
      "print m.count"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "4\n",
        "3\n",
        "2\n"
       ]
      }
     ],
     "prompt_number": 87
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Refresher on the normal uses of del - a built in python function which \n",
      "# acts on lists and on object attributes.\n",
      "\n",
      "class A:\n",
      "    pass\n",
      "\n",
      "a = A()\n",
      "a.x = 100\n",
      "\n",
      "del a.x\n",
      "# print a.x  # won't work anymore because we have deleted that attribute\n",
      "\n",
      "mylist = [1,2,3]\n",
      "del mylist[1]   # delete the '2'\n",
      "print mylist"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[1, 3]\n"
       ]
      }
     ],
     "prompt_number": 90
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Iterators"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Iteratos not only are a way of looping through a range of numbers, but also through lists etc.  \n",
      "\n",
      "Ultimately you want to be able to iterate through any collection - especially your own custom domain models."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for i in range(1,10):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 2 3 4 5 6 7 8 9\n"
       ]
      }
     ],
     "prompt_number": 92
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for s in ['red', 'blue', 'green']:\n",
      "    print s.upper()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "RED\n",
        "BLUE\n",
        "GREEN\n"
       ]
      }
     ],
     "prompt_number": 93
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print type(range(1,10))\n",
      "print type(xrange(1,10))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<type 'list'>\n",
        "<type 'xrange'>\n"
       ]
      }
     ],
     "prompt_number": 96
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# want to generate a list of 'squares' e.g. [1, 4, 9, 16, 25]\n",
      "class Squares(object):\n",
      "    def __init__(self, start, stop):\n",
      "        self.start = start\n",
      "        self.stop = stop\n",
      "    \n",
      "    def next(self):\n",
      "        if self.start > self.stop:\n",
      "            raise StopIteration\n",
      "        result = self.start  # could use a separate 'self.current'\n",
      "        self.start += 1\n",
      "        return result*result\n",
      "    \n",
      "    def __iter__(self):  # Marker to python that this is an iterator\n",
      "        return self    # could potentially return a different object\n",
      "    \n",
      "# Test\n",
      "\n",
      "for i in Squares(1,30):\n",
      "    print i,\n",
      "    \n",
      "# assert Squares(1,5) == [1, 4, 9, 16, 25]   # fails cos different objects really - probably could make a __cmp__ method if you are keen\n",
      "assert list(Squares(1,5)) == [1, 4, 9, 16, 25]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        " 1 4 9 16 25 36 49 64 81 100 121 144 169 196 225 256 289 324 361 400 441 484 529 576 625 676 729 784 841 900\n"
       ]
      }
     ],
     "prompt_number": 104
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Re-using iterators\n",
      "\n",
      "http://stackoverflow.com/questions/3266180/can-iterators-be-reset-in-python\n",
      "    \n",
      "No. Python's iterator protocol is very simple, and only provides one single method (.next() or __next__()), and no method to reset an iterator in general.\n",
      "\n",
      "The common pattern is to instead create a new iterator using the same procedure again.\n",
      "\n",
      "If you want to \"save off\" an iterator so that you can go back to its beginning, you may also fork the iterator by using itertools.tee\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# sq = Squares(1,10)  # an instance of an iterator\n",
      "# for i in sq:\n",
      "#     print i,\n",
      "# for i in sq:   # can we iterate twice?\n",
      "#     print i,\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 105
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "random iterator"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import random\n",
      "\n",
      "class random_iterator(object):\n",
      "    def __init__(self, num_values):\n",
      "        self.num_values = num_values\n",
      "    def next(self):\n",
      "        if self.num_values <= 0:\n",
      "            raise StopIteration\n",
      "        self.num_values -= 1\n",
      "        return random.randint(1,10)\n",
      "    def __iter__(self):\n",
      "        return self\n",
      "for i in random_iterator(12):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "7 1 6 9 4 5 10 4 8 7 6 10\n"
       ]
      }
     ],
     "prompt_number": 106
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Another version which takes the range you want.\n",
      "# YOU CAN HAVE ANY NUMBER OF PARAMTERS TO YOUR CUSTOM ITERATOR :-)\n",
      "import random\n",
      "\n",
      "class random_iterator(object):\n",
      "    def __init__(self, num_values, low_end, high_end):\n",
      "        self.num_values = num_values\n",
      "        self.low_end = low_end\n",
      "        self.high_end = high_end\n",
      "    def next(self):\n",
      "        if self.num_values <= 0:\n",
      "            raise StopIteration\n",
      "        self.num_values -= 1\n",
      "        return random.randint(self.low_end, self.high_end)\n",
      "    def __iter__(self):\n",
      "        return self\n",
      "for i in random_iterator(12,1000,2000):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1269 1596 1882 1664 1587 1761 1362 1601 1706 1790 1287 1219\n"
       ]
      }
     ],
     "prompt_number": 108
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Duplicates will be generated!\n",
      "for i in random_iterator(10,1,5):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "2 1 3 1 5 5 5 2 5 1\n"
       ]
      }
     ],
     "prompt_number": 110
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import random\n",
      "\n",
      "class unique_random_iterator(object):\n",
      "    def __init__(self, num_values, low_end, high_end):\n",
      "        self.num_values = num_values\n",
      "        self.low_end = low_end\n",
      "        self.high_end = high_end        \n",
      "        self.already_allocated = []\n",
      "    # Alternative approach perhaps...\n",
      "#         self.to_allocate = random.sample(xrange(1,numvalues))\n",
      "    def next(self):\n",
      "        if self.num_values <= 0:\n",
      "            raise StopIteration\n",
      "        self.num_values -= 1\n",
      "        current = random.randint(self.low_end, self.high_end)\n",
      "        max_tries = self.num_values # simple failsafe\n",
      "        while current in self.already_allocated:\n",
      "            current = random.randint(self.low_end, self.high_end)  # TODO FIX if too many duplicates and run out of ...\n",
      "            max_tries -= 1\n",
      "            if max_tries < 0:\n",
      "                self.already_allocated = []  # reset\n",
      "                return None\n",
      "        self.already_allocated.append(current)\n",
      "        return current\n",
      "    def __iter__(self):\n",
      "        return self\n",
      "for i in unique_random_iterator(45, 1, 5):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "5 1 4 3 2 None 3 1 4 5 2 None 5 3 4 1 2 None 2 3 1 4 5 None 1 4 5 2 3 None 4 1 2 3 5 None 3 4 2 1 None 3 2 1 4\n"
       ]
      }
     ],
     "prompt_number": 119
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** TIPS **\n",
      "\n",
      " - Use iterators to insulate code from the internals of your class.\n",
      " - Rather than an external class poking around in the private areas of your class and trying to find objects to iterate over, you instead create a nice 'iterator' interface which stops other code 'poking inside your class' \n",
      " - **Any** class can be an iterator.  It can do its normal functionality and **also** be an iterator (just by implementing the `next()` and `__iter__()` methods.\n",
      " - *Collection* classes are classes that own other smaller objects.  E.g. A company has many employees.  You typically make Collection classes iterators."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "CompanyIterator - iterating through your own domain model"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Super nice examples which show how to have a separate iterator class\n",
      "# which is utilised by the collection class (Company).\n",
      "# And shows that iterators can be generated on the fly for multiple \n",
      "# iterations\n",
      "class Company:\n",
      "    def __init__(self):\n",
      "        self.employees = []\n",
      "    def __iter__(self):\n",
      "        return CompanyIterator(company=self)\n",
      "    \n",
      "class Employee:\n",
      "    def __init__(self, name):\n",
      "        self.name = name\n",
      "    def __str__(self):\n",
      "        return \"Employee: \" + self.name\n",
      "\n",
      "class CompanyIterator(object):\n",
      "    def __init__(self, company):\n",
      "        self.current = 0\n",
      "        self.company = company\n",
      "    def next(self):\n",
      "        self.current += 1\n",
      "        if self.current > len(self.company.employees):\n",
      "            raise StopIteration\n",
      "        result = self.company.employees[self.current - 1]\n",
      "        return result\n",
      "                    \n",
      "c = Company()\n",
      "c.employees.append(Employee(\"Fred\"))\n",
      "c.employees.append(Employee(\"Mary\"))\n",
      "\n",
      "for employee in c:\n",
      "    print employee"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Employee: Fred\n",
        "Employee: Mary\n"
       ]
      }
     ],
     "prompt_number": 133
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for employee in c:\n",
      "    print employee\n",
      "for employee in c:\n",
      "    print employee"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Employee: Fred\n",
        "Employee: Mary\n",
        "Employee: Fred\n",
        "Employee: Mary\n"
       ]
      }
     ],
     "prompt_number": 135
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Generators"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Generator functions allow you to declare a function that behaves like an iterator i.e. it can be used in a for loop.\n",
      "\n",
      "Its a simpler way.\n",
      "\n",
      "The simplification of code is a result of generator function and generator expression support provided by Python.\n",
      "\n",
      "The problem with iterators:\n",
      " - there is a lot of boilerplate\n",
      " - the logic has to be expressed in a somewhat convoluted way\n",
      "\n",
      "Python provides generator functions as a convenient shortcut to building iterators"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def squares(start, stop):\n",
      "    for i in range(start, stop):\n",
      "        yield i*i\n",
      "\n",
      "for i in squares(1,10):\n",
      "    print i,"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1 4 9 16 25 36 49 64 81\n"
       ]
      }
     ],
     "prompt_number": 137
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Think of `yield` as a return statement, but which keeps the function alive**\n",
      "\n",
      "Cannot have yield and return in the same function"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def fun():\n",
      "    yield \"we are having fun\"\n",
      "    my_dict = {'a':100, 'b':200}\n",
      "    yield my_dict\n",
      "    yield 100\n",
      "    yield 2\n",
      "    for key in my_dict.keys():\n",
      "        yield key\n",
      "    yield [1,2,3,4,5]\n",
      "\n",
      "for info in fun():\n",
      "    print info"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "we are having fun\n",
        "{'a': 100, 'b': 200}\n",
        "100\n",
        "2\n",
        "a\n",
        "b\n",
        "[1, 2, 3, 4, 5]\n"
       ]
      }
     ],
     "prompt_number": 156
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gen_stuff = fun()\n",
      "print type(gen_stuff)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<type 'function'>\n",
        "<type 'generator'>\n"
       ]
      }
     ],
     "prompt_number": 157
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "gen_stuff.next()   # Can call this repeatedly"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 154,
       "text": [
        "'we are having fun'"
       ]
      }
     ],
     "prompt_number": 154
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# independent iterator objects each keep their own state !\n",
      "fun_times = [fun(), fun(), fun()]\n",
      "print fun_times[0].next()  # should be 'we are having fun'\n",
      "print fun_times[0].next()  # first generator should be up to {'a': 100, 'b': 200}\n",
      "print fun_times[1].next()  # should also be 'we are having fun'\n",
      "print fun_times[0].next()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "we are having fun\n",
        "{'a': 100, 'b': 200}\n",
        "we are having fun\n",
        "100\n"
       ]
      }
     ],
     "prompt_number": 159
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# pretty sophisticated generator\n",
      "\n",
      "def sorted_dict_iterator(dictionary):\n",
      "    \n",
      "    # First solution\n",
      "    \n",
      "    keys = dictionary.keys()\n",
      "    keys.sort()\n",
      "    i = 0\n",
      "    for key in keys:\n",
      "        yield key, dictionary[key], i\n",
      "        i += 1\n",
      "    \n",
      "    # or\n",
      "    \n",
      "    for i, key in enumerate(sorted(dictionary)):  # sorted copy of dictionary\n",
      "        yield key, dictionary[key], i\n",
      "\n",
      "    # or\n",
      "    \n",
      "    for key in keys:\n",
      "        yield key, dictionary[key], keys.index(key)\n",
      "        \n",
      "d = {'z' : 'zz',\n",
      "     'm' : 'mm',\n",
      "     'a' : 'aa'}\n",
      "for key, value, index in sorted_dict_iterator(d):\n",
      "    print 'key %s has value %s and index is %d' % (key, value, index)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "key a has value aa and index is 0\n",
        "key m has value mm and index is 1\n",
        "key z has value zz and index is 2\n",
        "key a has value aa and index is 0\n",
        "key m has value mm and index is 1\n",
        "key z has value zz and index is 2\n",
        "key a has value aa and index is 0\n",
        "key m has value mm and index is 1\n",
        "key z has value zz and index is 2\n"
       ]
      }
     ],
     "prompt_number": 164
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Unit Testing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Unit testing automates the comparing the behaviour of your code with expected results"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import unittest\n",
      "\n",
      "class TestAAAA(unittest.TestCase):\n",
      "    \n",
      "    def test_1(self):\n",
      "        self.assertTrue(True)\n",
      "        self.assertFalse(False)\n",
      "        self.assertEqual(2, 1+1)\n",
      "        \n",
      "    def test_math_stuff(self):\n",
      "        \"\"\"\n",
      "        This tests that pi and other math functions work\n",
      "        \"\"\"\n",
      "        import math\n",
      "        self.assertEqual(3, int(math.pi))\n",
      "\n",
      "suite = unittest.TestLoader().loadTestsFromTestCase(TestAAAA)\n",
      "unittest.TextTestRunner(verbosity=2).run(suite)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "test_1 (__main__.TestAAAA) ... "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "ok\n",
        "test_math_stuff (__main__.TestAAAA) ... "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "ok\n",
        "\n",
        "----------------------------------------------------------------------\n",
        "Ran 2 tests in 0.002s\n",
        "\n",
        "OK\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 166,
       "text": [
        "<unittest.runner.TextTestResult run=2 errors=0 failures=0>"
       ]
      }
     ],
     "prompt_number": 166
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Tests are and should be independent from one another - one test should not affect any other test.\n",
      "\n",
      "import unittest\n",
      "\n",
      "class TestB(unittest.TestCase):\n",
      "    \n",
      "    @classmethod\n",
      "    def setUpClass(cls):  # Only runs once for the whole class\n",
      "        print 'SETUP CLASSSSSSS!', cls\n",
      "        \n",
      "    def setUp(self):  # Special name of unit test set up - runs before EVERY test\n",
      "        print 'SETUP RUN'\n",
      "        self.my_array = [1,2,3,4,5]\n",
      "\n",
      "    def tearDown(self):  # Special name\n",
      "        print 'TEARDOWN RUN'\n",
      "        self.my_array = None\n",
      "\n",
      "    def test_adel(self):\n",
      "        del self.my_array[1]\n",
      "        self.assertEqual(4, len(self.my_array))\n",
      "        \n",
      "    def check_length(self):  # utility \n",
      "        self.assertEqual(5, len(self.my_array))\n",
      "        \n",
      "    def test_count(self):\n",
      "#         self.assertEqual(5, len(self.my_array))\n",
      "        self.check_length()\n",
      "\n",
      "    def test_count_again(self):\n",
      "        self.my_array.pop()\n",
      "        self.my_array.append(99)\n",
      "        self.check_length()\n",
      "    \n",
      "suite = unittest.TestLoader().loadTestsFromTestCase(TestB)\n",
      "unittest.TextTestRunner(verbosity=2).run(suite)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "test_adel (__main__.TestB) ... "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "ok\n",
        "test_count (__main__.TestB) ... "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "ok\n",
        "test_count_again (__main__.TestB) ... "
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "SETUP CLASSSSSSS! <class '__main__.TestB'>\n",
        "SETUP RUN\n",
        "TEARDOWN RUN\n",
        "SETUP RUN\n",
        "TEARDOWN RUN\n",
        "SETUP RUN\n",
        "TEARDOWN RUN\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stderr",
       "text": [
        "ok\n",
        "\n",
        "----------------------------------------------------------------------\n",
        "Ran 3 tests in 0.003s\n",
        "\n",
        "OK\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 179,
       "text": [
        "<unittest.runner.TextTestResult run=3 errors=0 failures=0>"
       ]
      }
     ],
     "prompt_number": 179
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Command-Line Interface to unit testing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Command-Line Interface, can be used to run modules and sub packages. The unittest module can be used from the command line to run tests from modules, classes or even individual test methods."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "EXERCISE: create two modules and run the tests\n",
      "\n",
      "    python -m unittest test_module1 test_module2\n",
      "    \n",
      "EXERCISE: target running all tests in a class\n",
      "\n",
      "    python -m unittest test_module.TestClass\n",
      "\n",
      "EXERCISE: target running one specific unit test\n",
      "\n",
      "    python -m unittest test_module.TestClass.test_method"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Benefit of command line invocation is that you **don't need the suite=....** bootstrapping stuff in your unit test code.  Also it allows you to target running specifica test classes, or test methods"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Other cool commands:\n",
      "\n",
      "    # List options e.g. quiet mode\n",
      "    python -m unittest --help\n",
      "\n",
      "    # will search for tests in current directory - you don't \n",
      "    # have to specify the module in this case\n",
      "    python -m unittest discover\n",
      "    python -m unittest discover -v\n",
      "    \n",
      "    # can specify pattern matching for the files to search in\n",
      "    python -m unittest discover -p 'test*.py'"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "More official doco on python unit testing https://docs.python.org/2/library/unittest.html"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Setting up a good project structure"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "See example directory \n",
      "\n",
      "    testing_example2 \n",
      "    \n",
      "re how to set up a good project structure\n",
      "\n",
      "Needs good knowledge of modules and python module mode\n",
      "\n",
      "    python -m .....\n",
      "    \n",
      "    "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "    (py2k)Andys-MacBook-Air:testing_example2 Andy$ tree\n",
      "    .\n",
      "    \u251c\u2500\u2500 lib\n",
      "    \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n",
      "    \u2502\u00a0\u00a0 \u251c\u2500\u2500 beamline_stuff\n",
      "    \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n",
      "    \u2502\u00a0\u00a0 \u2502\u00a0\u00a0 \u2514\u2500\u2500 beamline_calc.py\n",
      "    \u2502\u00a0\u00a0 \u251c\u2500\u2500 light_calculator.py\n",
      "    \u2502\u00a0\u00a0 \u2514\u2500\u2500 water_calc\n",
      "    \u2502\u00a0\u00a0     \u251c\u2500\u2500 __init__.py\n",
      "    \u2502\u00a0\u00a0     \u251c\u2500\u2500 water1.py\n",
      "    \u2502\u00a0\u00a0     \u2514\u2500\u2500 water2.py\n",
      "    \u251c\u2500\u2500 main.py\n",
      "    \u251c\u2500\u2500 readme.txt\n",
      "    \u251c\u2500\u2500 scripts\n",
      "    \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n",
      "    \u2502\u00a0\u00a0 \u2514\u2500\u2500 util1.py\n",
      "    \u251c\u2500\u2500 settings.py\n",
      "    \u2514\u2500\u2500 tests\n",
      "        \u251c\u2500\u2500 __init__.py\n",
      "        \u251c\u2500\u2500 speed_tests.py\n",
      "        \u251c\u2500\u2500 tests1.py\n"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Logging"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You (typically) **don't put print statements anywhere** (unless you are debugging).  \n",
      "\n",
      "Return values from your functions and only at the top level of your application you would print out final values or save those values to files or generate html files.\n",
      "\n",
      "http://www.shutupandship.com/2012/02/how-python-logging-module-works.html\n",
      "\n",
      "  - Handlers are the actual places we output to.\n",
      "  - Loggers are the named 'bucket' objects we create and write to in our code.\n",
      "  - There can be many handlers attached to one logger.\n",
      "  - Messages sent to Loggers propagate to parent loggers (based on . syntax with '' root logger at the top) unless you supress.\n",
      "  - Error level on both the logger and handler will prevent messages of a lower value getting through (lowest is debug, then info, error is a high value)\n",
      "\n",
      "Official doco https://docs.python.org/2/library/logging.html"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "These are the logging levels.\n",
      "\n",
      "    CRITICAL\n",
      "    ERROR\n",
      "    WARNING\n",
      "    INFO\n",
      "    DEBUG\n",
      "    NOTSET"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "more next week..."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Homework"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Create your own logging system which implements\n",
      "\n",
      "    log.info(msg)\n",
      "    log.error(msg)\n",
      "    \n",
      "and each message should be output (appended) to a file. \n",
      "\n",
      "Also each message should have a date and time prepended to it."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}