Advanced-Python.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "a54ce074-c06d-491e-87ea-edb7641f9fb1",
      "metadata": {
        "tags": [],
        "id": "a54ce074-c06d-491e-87ea-edb7641f9fb1"
      },
      "source": [
        "# Advanced Python - Part 1 <img src=\"/content/drive/MyDrive/python4hpc-main/images/advanced/python-logo.png\" style=\"align: right; width: 220px\" />\n",
        "\n",
        "Before we explore the more advanced toolkits and libraries we first want to give you an overview of topics such as\n",
        "- built-in python collections and when to use them\n",
        "- deep dive into object-oriented programming behind the scenes,\n",
        "- what are the functional programming parts of python and how to use them\n",
        "- a few words about the garbage collector and finally\n",
        "- tools python has for simple caching strategies\n",
        "\n",
        "## Why should I care?\n",
        "\n",
        "Why should I care about basic python stuff? I'm using framework xyz ... why is it important?\n",
        "\n",
        "<div class=\"alert alert-block alert-info\"><b>\n",
        "Knowing how to better structure your code, what tools are provided and what performance impact they have: it really matters, especially when you are writing highly complex code that should perform well.<br/>\n",
        "Only once you understand how the environment works, what it does behind the scenes, only then you can take full control of it.\n",
        "</b></div>\n",
        "\n",
        "## Python version and implementation\n",
        "\n",
        "Unless noted otherwise the performance details and the implementation notes refer to the latest __Python 3.10__ version as well as the official __CPython__ interpreter."
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i2aI2ay_URN9",
        "outputId": "282a3b7a-5990-424c-b3e7-80f22e44109d"
      },
      "id": "i2aI2ay_URN9",
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "id": "682d1046-c351-4138-b113-656d1e9904ed",
      "metadata": {
        "tags": [],
        "id": "682d1046-c351-4138-b113-656d1e9904ed"
      },
      "source": [
        "## Containers and Utility classes\n",
        "\n",
        "Python already comes with some very sophisticated built-in collection types everybody should know about\n",
        "\n",
        "- `tuple` - fixed size list of arbitrary items\n",
        "- `list` - simple dynamic list of arbitrary items\n",
        "- `set` - unordered collection of hashable values that supports mathematical operations\n",
        "- `frozenset` - same as set but immutable and thus hashable\n",
        "- `dict` - the default mapping type for key-value storage\n",
        "\n",
        "There are also more advanced lesser known collection types and helpful utilities within the python library\n",
        "\n",
        "Overview of the classes in the _collection_ package:\n",
        "\n",
        "- `namedtuple` - a factory function to create tuple subclasses with named fields\n",
        "- `deque` - a double-ended queue\n",
        "- `ChainMap` - a proxy class to provide a common view to multiple mapping instances\n",
        "- `Counter` - simple helper class for counting hashable objects\n",
        "- `OrderedDict` - this dict implementation focuses on insertion order and is optimised for reordering\n",
        "- `defaultdict` - dict that provides default values for keys\n",
        "- `UserDict`, `UserList`, `UserStr` - base classes for implementations of dict, list or str subclasses\n",
        "\n",
        "Another important container type is the `dataclass` from the _dataclasses_ packages.\n",
        "\n",
        "Also have a look at the following pages in the python documentation and the python wiki:\n",
        "* __[Sequence types](https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range)__\n",
        "* __[Collection package docs](https://docs.python.org/3/library/collections.html)__\n",
        "* __[TimeComplexity of Python Collections](https://wiki.python.org/moin/TimeComplexity)__\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "75744858-f7b6-49bf-a190-f9d1e5a0e345",
      "metadata": {
        "id": "75744858-f7b6-49bf-a190-f9d1e5a0e345"
      },
      "source": [
        "### Everyday containers"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "66bf4887",
      "metadata": {
        "id": "66bf4887"
      },
      "source": [
        "#### [Tuple](https://docs.python.org/3/library/stdtypes.html#tuple)\n",
        "\n",
        "A `tuple` is a simple immutable fixed-size sequence of arbitrary values.\n",
        "\n",
        "It is often used implicitly as return values in functions or in general when the number of elements is fixed and ease of use is important.\n",
        "\n",
        "Since tuples are immutable they work with `hash` and have a fixed hash value. Because of this they can be used in all cases where an immutable hashable datatype is required (e.g. keys in dicts)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0ccfa0d9",
      "metadata": {
        "tags": [],
        "id": "0ccfa0d9"
      },
      "outputs": [],
      "source": [
        "# parentheses are actually optional but often used for clarity\n",
        "a_tuple = (1, 'string', 32, )\n",
        "print('via parentheses:', a_tuple)\n",
        "\n",
        "# this is also called 'packing'\n",
        "a_tuple = 1, 'string', 32\n",
        "print('packing:', a_tuple)\n",
        "\n",
        "# attention: if parentheses are used like this a 'generator object' will be constructed instead\n",
        "# see \"Functional Programming\" for more info\n",
        "print('generator: ', (a for a in a_tuple))\n",
        "\n",
        "# alternative creation via tuple constructor using an iterable/sequence\n",
        "a_tuple = tuple([1, 'string', 32, ])\n",
        "print('via constructor: ', a_tuple)\n",
        "\n",
        "# deconstruction of a tuple into values aka 'unpacking'\n",
        "a, b, c = a_tuple\n",
        "print('unpacked:', a, b, c)\n",
        "\n",
        "# indexed access\n",
        "print('indexed access:', f'{a_tuple[0]}, {a_tuple[2]}')\n",
        "\n",
        "# tuples are implicitly used to return multiple values from a function\n",
        "def calc():\n",
        "    return 1, 2, 3\n",
        "\n",
        "print('tuple from return:', calc())\n",
        "\n",
        "# tuples can be hashed (if they only contain hashable objects)\n",
        "print('tuple hashed: ', hash(a_tuple))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b3c685ce",
      "metadata": {
        "id": "b3c685ce"
      },
      "source": [
        "#### [List](https://docs.python.org/3/tutorial/datastructures.html#more-on-lists)\n",
        "\n",
        "The default python collection for storing items in a consecutive list. Lists are mutable and thus can dynamically grow and shrink.\n",
        "\n",
        "Contrary to the name 'list' the default python implementation actually uses an array of references internally and not e.g. a linked list. In general the python implementation always accesses object instances indirectly by only holding references to the actual object. This is necessary to allow for arbitrary types in almost all places. Coming back to the list this means that the largest runtime costs come from growing beyond the current size and inserting/deleting elements at the start. In these cases the underlying array of references needs to be resized (realloc + copy) or contained references need to be moved to make or remove space for the inserted/deleted element.\n",
        "\n",
        "A list can easily be used as a stack by using `append` and `pop` methods but if you need e.g. a FIFO queue other types are preferable (see `deque`). The same goes for frequent lookups where other datatypes are more performant e.g. `set` or `dict`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "acf0b4de",
      "metadata": {
        "id": "acf0b4de"
      },
      "outputs": [],
      "source": [
        "# use square brackets and comma separated values to create a list and assign it to a variable\n",
        "b_list = ['Hello', 'World', '!']\n",
        "print('via square brackets:', b_list)\n",
        "\n",
        "# alternative creation via constructor using an iterable e.g. here from a tuple\n",
        "b_list = list(('Hello', 'World', '!'))\n",
        "print('via list constructor:', b_list)\n",
        "\n",
        "# deconstructiong a list into variables works too\n",
        "a, b, c = b_list\n",
        "print('a, b, c: ', a, b, c)\n",
        "\n",
        "# insert element at index\n",
        "b_list.insert(2, 'Johnny')\n",
        "print('after insert:', b_list)\n",
        "\n",
        "# append to the end\n",
        "b_list.append('!')\n",
        "print('after append:', b_list)\n",
        "\n",
        "# indexed access of elements\n",
        "print('indexed access:', f'{b_list[1]} {b_list[4]}')\n",
        "\n",
        "# remove item by using del & index\n",
        "del b_list[2]\n",
        "print('b_list after del:', b_list)\n",
        "\n",
        "# remove item by value\n",
        "b_list.remove('World')\n",
        "print('b_list after remove:', b_list)\n",
        "\n",
        "# remove last item\n",
        "b_list.pop()\n",
        "print('b_list after pop:', b_list)\n",
        "\n",
        "# copy whole list\n",
        "c_list = b_list.copy()\n",
        "print('c_list (copy):', c_list)\n",
        "\n",
        "# reverse list\n",
        "c_list.reverse()\n",
        "print('c_list (reversed):', c_list)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "077309d5",
      "metadata": {
        "id": "077309d5"
      },
      "source": [
        "#### [Set and Frozenset](https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset)\n",
        "\n",
        "If the uniqueness of items needs to be guaranteed and/or set operations such as `union`, `intersection` etc are needed, the `set` collection is the datatype of choice. In addition membership tests are also usually cheap to perform. Be aware that `set` is *unordered* so the insertion order is not guaranteed.\n",
        "\n",
        "Internally a `set` can be thought of as an unordered hashmap. This means that the items are inserted based on their hash value. Values that have been added to a set are required to not change their hash once they have been added. Breaking this rule breaks the internal mechanics of the set and produces possibly undefined behaviour.\n",
        "\n",
        "Sets can also be used with operators such as '&' (intersection), '|' (union), '-' (difference) and '^' (symmetric_difference) instead of calling the method.\n",
        "\n",
        "A `frozenset` is basically the same as `set` but it is immutable and hashable. Once created its contents cannot be changed anymore. These additional constraints allow the use of such instances as keys for e.g. a dictionary. Regarding performance both types are equivalent."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3da77c03",
      "metadata": {
        "id": "3da77c03"
      },
      "outputs": [],
      "source": [
        "# create two sets with chars from strings (=iterable)\n",
        "a_set = set('Hello World!'.upper())\n",
        "print(a_set)\n",
        "\n",
        "b_set = set('How are you today?'.upper())\n",
        "print(b_set)\n",
        "\n",
        "# get characters in both sets\n",
        "elements_in_both = a_set.intersection(b_set)\n",
        "print('elements in both sets: ', elements_in_both)\n",
        "\n",
        "# get differences (not symmetrical)\n",
        "elements_in_a_but_not_in_b = a_set.difference(b_set)\n",
        "print('elements in a but not in b: ', elements_in_a_but_not_in_b)\n",
        "elements_in_b_but_not_in_a = b_set.difference(a_set)\n",
        "print('elements in b but not in a: ', elements_in_b_but_not_in_a)\n",
        "\n",
        "# get characters that are only in one set\n",
        "elements_only_in_one = a_set.symmetric_difference(b_set)\n",
        "print('elements only in one set: ', elements_only_in_one)\n",
        "\n",
        "# all unique elements\n",
        "all_elements = a_set.union(b_set)\n",
        "print('all elements: ', all_elements)\n",
        "\n",
        "# demonstrate that e.g. lists cannot be added to a set\n",
        "try:\n",
        "    set(([1, 2, 3], [4, 5, 6]))\n",
        "except Exception as e:\n",
        "    print('Cannot add unhashable items - ', e)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ce60b194",
      "metadata": {
        "id": "ce60b194"
      },
      "source": [
        "#### [Dict](https://docs.python.org/3/library/stdtypes.html#mapping-types-dict)\n",
        "\n",
        "Dictionaries are a very central datatype in python. It stores unique keys that point to a certain value.\n",
        "\n",
        "Lookups via keys is usually fast - this means that membership tests & retrieving a value is considerably cheap."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2bca66c2",
      "metadata": {
        "id": "2bca66c2"
      },
      "outputs": [],
      "source": [
        "# initialise from kwargs\n",
        "a_dict = dict(a=1, b=2)\n",
        "\n",
        "# initialise with {}\n",
        "a_dict = {'a': 1, 'b': 2}\n",
        "print(a_dict)\n",
        "\n",
        "# update dict from another dict\n",
        "a_dict.update({'b': 3, 'c': 4})\n",
        "print(a_dict)\n",
        "\n",
        "# insert a new key-value pair\n",
        "a_dict['d'] = 5\n",
        "\n",
        "# change an existing value\n",
        "a_dict['a'] -= 1\n",
        "print(a_dict)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "8d2560bd-cffd-4ab7-94ed-98905fbf99d0",
      "metadata": {
        "id": "8d2560bd-cffd-4ab7-94ed-98905fbf99d0"
      },
      "source": [
        "As stated before a key needs to be hashable and should always produce the same hash so that it can be used in a `dict` without problems.\n",
        "\n",
        "A `dict` instance itself is not hashable in general which means that dict instances cannot directly be used as keys.\n",
        "\n",
        "However you can easily have a `dict` value in a class instance and use the instance as key. This works because classes have a default hash implementation that uses the actual memory address for computing the hash. Please note that this also means that to instances with the same values will have a different hash, since they are not the same object!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a262f7c4-98f1-4125-ab53-ea92caa87cfe",
      "metadata": {
        "tags": [],
        "id": "a262f7c4-98f1-4125-ab53-ea92caa87cfe"
      },
      "outputs": [],
      "source": [
        "a_dict = dict(a=1, b=2)\n",
        "\n",
        "# demonstrate that dicts are not hashable\n",
        "try:\n",
        "    hash(a_dict)\n",
        "except TypeError as e:\n",
        "    print('Cannot hash dict - ', e)\n",
        "\n",
        "# demonstrate that dicts cannot have dicts as keys\n",
        "dict_key = dict(z=100)\n",
        "try:\n",
        "    a_dict[dict_key] = 3\n",
        "except TypeError as e:\n",
        "    print('Cannot have dict as key - ', e)\n",
        "\n",
        "# however: we can have classes containing dicts as keys\n",
        "class MyClass: pass\n",
        "obj = MyClass()\n",
        "obj.my_attrib = dict(y=101)\n",
        "\n",
        "a_dict[obj] = 4\n",
        "print(a_dict)\n",
        "\n",
        "# the object have a different hash\n",
        "obj2 = MyClass()\n",
        "obj2.my_attrib = obj.my_attrib\n",
        "print('hash obj:', hash(obj))\n",
        "print('hash obj2:', hash(obj2))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0b4cd2ee",
      "metadata": {
        "id": "0b4cd2ee"
      },
      "source": [
        "#### [Deque](https://docs.python.org/3/library/collections.html#deque-objects)\n",
        "\n",
        "The name `deque` stands for *double ended queue* and already tells a lot about the desired use case of this class. If there is the need to quickly insert and remove from the beginning or end of the data structure this collection can be used (FIFO / LIFO). It provides the necessary methods and is optimised for this very purpose.\n",
        "\n",
        "In addition to that a `deque` can also be used with a `maxlen` argument. This means that when new items are added to one side, the collection discards as many items from the other side to satisfy the `maxlen` count again."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "36ccb148",
      "metadata": {
        "id": "36ccb148"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "a_deque = deque()\n",
        "\n",
        "# append on the right side\n",
        "a_deque.append(1)\n",
        "\n",
        "# append single element on the left side\n",
        "a_deque.appendleft(10)\n",
        "print('after append right and left:', a_deque)\n",
        "\n",
        "# extend from iterable on the left side\n",
        "# note: adds the iterable in reverse\n",
        "a_deque.extendleft((4, 5, 6, ))\n",
        "print('after extend left:', a_deque)\n",
        "\n",
        "# pop an item from the left isde\n",
        "a_deque.popleft()\n",
        "\n",
        "# pop an item from the right side\n",
        "a_deque.pop()\n",
        "print('after pop left and pop right:', a_deque)\n",
        "\n",
        "# rotate elements right by 1 element\n",
        "a_deque.rotate(1)\n",
        "print('after rotate right by 1:', a_deque)\n",
        "\n",
        "# create a deque with maxlength of 3\n",
        "b_deque = deque(maxlen=3)\n",
        "b_deque.extend((1, 2, 3, ))\n",
        "print('deque with maxlen:', b_deque)\n",
        "\n",
        "# appending on the right side shifts out an element from the beginning\n",
        "b_deque.append(4)\n",
        "print('append element:', b_deque)\n",
        "\n",
        "# appending on the left side shifts out an element from the end\n",
        "b_deque.appendleft(10)\n",
        "print('append left:', b_deque)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "61438c9f-429e-4835-81d8-e60f0ff20e8e",
      "metadata": {
        "tags": [],
        "id": "61438c9f-429e-4835-81d8-e60f0ff20e8e"
      },
      "source": [
        "Find a simple performance comparison between *list* & *deque* below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "96fabbbe-8d03-453e-8667-361d008f800d",
      "metadata": {
        "tags": [],
        "id": "96fabbbe-8d03-453e-8667-361d008f800d"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "def lifo_list():\n",
        "    l = []\n",
        "    for i in range(1000):\n",
        "        l.append(i)\n",
        "    while l:\n",
        "        item = l.pop()\n",
        "\n",
        "print('LIFO with list')\n",
        "%timeit -n 100 -r 5 lifo_list()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "24bafc5b-127c-49ca-a1e2-78fa64557cb6",
      "metadata": {
        "id": "24bafc5b-127c-49ca-a1e2-78fa64557cb6"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "def lifo_deque():\n",
        "    d = deque()\n",
        "    for i in range(1000):\n",
        "        d.append(i)\n",
        "    while d:\n",
        "        item = d.pop()\n",
        "\n",
        "print('LIFO with deqeue')\n",
        "%timeit -n 100 -r 5 lifo_deque()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d5f3775a-c692-4faa-bb90-b6452e5e0898",
      "metadata": {
        "id": "d5f3775a-c692-4faa-bb90-b6452e5e0898"
      },
      "source": [
        "Used as a _LIFO_ (last in first out) the performance is nearly the same."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8e33f2be-588d-4bef-ba06-6fde386049ba",
      "metadata": {
        "tags": [],
        "id": "8e33f2be-588d-4bef-ba06-6fde386049ba"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "def fifo_list():\n",
        "    l = []\n",
        "    for i in range(1000):\n",
        "        l.append(i)\n",
        "    while l:\n",
        "        item = l[0]\n",
        "        del l[0]\n",
        "\n",
        "print('FIFO with list')\n",
        "%timeit -n 100 -r 5 fifo_list()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1260abd1-c86e-4519-97fc-ad6d19d4398b",
      "metadata": {
        "tags": [],
        "id": "1260abd1-c86e-4519-97fc-ad6d19d4398b"
      },
      "outputs": [],
      "source": [
        "from collections import deque\n",
        "\n",
        "def fifo_deque():\n",
        "    d = deque()\n",
        "    for i in range(1000):\n",
        "        d.append(i)\n",
        "    while d:\n",
        "        item = d.popleft()\n",
        "\n",
        "print('FIFO with deqeue')\n",
        "%timeit -n 100 -r 5 fifo_deque()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b43c7708-8398-4d22-96b7-4b362a4c9657",
      "metadata": {
        "id": "b43c7708-8398-4d22-96b7-4b362a4c9657"
      },
      "source": [
        "When used as a _FIFO_ (first in first out) the deque is definitely faster."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fae01b26",
      "metadata": {
        "id": "fae01b26"
      },
      "source": [
        "#### [OrderedDict](https://docs.python.org/3/library/collections.html#collections.OrderedDict)\n",
        "\n",
        "The main use case for `OrderedDict` is when insertion order is important and frequent reordering operations are necessary. Because of that focus the underlying implementation is very different from the default `dict`: internally a doubly linked list is used in CPython.\n",
        "\n",
        "The CPython `dict` implementation also guarantees the insertion order of keys since __[Python 3.6](https://docs.python.org/3.6/whatsnew/3.6.html#new-dict-implementation)__. This implementation detail of `dict` was officially added to the language spec in __[Python 3.7](https://docs.python.org/3.7/whatsnew/3.7.html#summary-release-highlights)__ and made the `OrderedDict` less important compared to earlier python version. Still it can be a very useful class for specific problems."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "881d5736",
      "metadata": {
        "id": "881d5736"
      },
      "outputs": [],
      "source": [
        "from collections import OrderedDict\n",
        "\n",
        "# create a new ordered dict form a dict argument\n",
        "o_dict = OrderedDict({'c': 10, 'a': 2, 'b': 3})\n",
        "print(o_dict.keys())\n",
        "\n",
        "# move the key 'c' to the end of the ordered dict\n",
        "o_dict.move_to_end('c')\n",
        "print(o_dict.keys())\n",
        "\n",
        "# remove an item from the back\n",
        "o_dict.popitem()\n",
        "print(o_dict.keys())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dbe971d5-4f38-4b87-b911-dfd1d3c49f35",
      "metadata": {
        "id": "dbe971d5-4f38-4b87-b911-dfd1d3c49f35"
      },
      "source": [
        "Although insertion order is guaranteed for `dict` it is still not easy to reorder elements e.g. moving an element to the back can be done but inserting at specific positions gets extremely tricky without recreating the whole dictionary. The `OrderedDict` has simple solutions for such use cases\n",
        "\n",
        "Consider this example: We have given input dictionary and want to move some elements to the back of the internal order."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85fc7f59-7776-4d6c-ab0f-aec41abaf2d4",
      "metadata": {
        "tags": [],
        "id": "85fc7f59-7776-4d6c-ab0f-aec41abaf2d4"
      },
      "outputs": [],
      "source": [
        "d = {i: i*2 for i in range(100000)}\n",
        "keys = [i*2 for i in range(10000)]\n",
        "\n",
        "def dict_move_keys_to_back_inplace():\n",
        "    for k in keys:\n",
        "        d[k] = d.pop(k)\n",
        "\n",
        "print(list(d.items())[:10])\n",
        "dict_move_keys_to_back_inplace()\n",
        "print(list(d.items())[:10])\n",
        "\n",
        "print('standard dict move elements to back')\n",
        "%timeit -n 100 -r 5 dict_move_keys_to_back_inplace()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "57774088-302f-45ae-a52e-b8efd4e0c9a0",
      "metadata": {
        "tags": [],
        "id": "57774088-302f-45ae-a52e-b8efd4e0c9a0"
      },
      "outputs": [],
      "source": [
        "from collections import OrderedDict\n",
        "\n",
        "d = OrderedDict({i: i*2 for i in range(100000)})\n",
        "keys = [i*2 for i in range(10000)]\n",
        "\n",
        "def ordered_dict_move_keys_to_back_inplace():\n",
        "    for k in keys:\n",
        "        d.move_to_end(k)\n",
        "\n",
        "print(list(d.items())[:10])\n",
        "ordered_dict_move_keys_to_back_inplace()\n",
        "print(list(d.items())[:10])\n",
        "\n",
        "print('ordered dict move elements to back')\n",
        "%timeit -n 100 -r 5 ordered_dict_move_keys_to_back_inplace()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "20429b43-dae4-457e-82e9-5f096a8997f1",
      "metadata": {
        "id": "20429b43-dae4-457e-82e9-5f096a8997f1"
      },
      "source": [
        "However the internal data structure of `OrderedDict` also has considerable drawbacks when space efficiency, iteration speed and update operations are of importance.\n",
        "\n",
        "In such cases a standard `dict` usually performs better - consider the following very simple insert example"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5c61f54e-89d7-42c8-87a0-effc7910cce9",
      "metadata": {
        "tags": [],
        "id": "5c61f54e-89d7-42c8-87a0-effc7910cce9"
      },
      "outputs": [],
      "source": [
        "def dict_insert():\n",
        "    d = dict()\n",
        "    for i in range(10000):\n",
        "        d[i] = i\n",
        "\n",
        "print('standard dict insert')\n",
        "%timeit -n 100 -r 5 dict_insert()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "60e8e6f3-6dc0-40c3-8859-f71be9754478",
      "metadata": {
        "tags": [],
        "id": "60e8e6f3-6dc0-40c3-8859-f71be9754478"
      },
      "outputs": [],
      "source": [
        "from collections import OrderedDict\n",
        "\n",
        "def ordered_dict_insert():\n",
        "    d = OrderedDict()\n",
        "    for i in range(10000):\n",
        "        d[i] = i\n",
        "\n",
        "print('ordered dict insert')\n",
        "%timeit -n 100 -r 5 ordered_dict_insert()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "76bf17bf-fbfc-45ae-9381-8b944fd07e4b",
      "metadata": {
        "tags": [],
        "id": "76bf17bf-fbfc-45ae-9381-8b944fd07e4b"
      },
      "source": [
        "### Utility Containers"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a9030272",
      "metadata": {
        "id": "a9030272"
      },
      "source": [
        "#### [Namedtuple](https://docs.python.org/3/library/collections.html#collections.namedtuple)\n",
        "\n",
        "Named tuples were primarily created to improve readability. Behind the scenes it is actually a factory function that creates a new type with the fields given as arguments and the `tuple` as base class. The implementation of the new type makes sure that all necessary values are there upon initialisation and that an access to a non-existing element raises an exception. It supports all the operations a regular `tuple` would plus access to the attributes by name.\n",
        "\n",
        "Regarding performance a `namedtuple` is usually worse than a regular `tuple` since there is additional code running to realise the checks behind the scenes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6a7d3ed0-2f44-40cd-80e1-7bbc75e4a224",
      "metadata": {
        "id": "6a7d3ed0-2f44-40cd-80e1-7bbc75e4a224"
      },
      "outputs": [],
      "source": [
        "from collections import namedtuple\n",
        "from typing import NamedTuple\n",
        "\n",
        "# create a named tuple by using the factory function `namedtuple`\n",
        "TupleFG = namedtuple('TupleFG', ('f', 'g'))\n",
        "print('Inheritance chain: ', TupleFG.mro())\n",
        "\n",
        "x = TupleFG(1, 2)\n",
        "print('Instance of TupleFG: ', x)\n",
        "\n",
        "# alternative way of creating a named tuple type\n",
        "# this way it is possible to easily add type hints for the IDE\n",
        "class TupleXY(NamedTuple):\n",
        "    x: int\n",
        "    y: int\n",
        "\n",
        "y = TupleXY(3, 4)\n",
        "print('Instance of TupleXY: ', y)\n",
        "\n",
        "# use the factory function to create a named tuple\n",
        "z = TupleXY._make((1, 2,))\n",
        "print('Instance of TupleXY via factory function: ', z)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "062481af",
      "metadata": {
        "id": "062481af"
      },
      "source": [
        "#### [Dataclasses](https://docs.python.org/3/library/dataclasses.html)\n",
        "\n",
        "Dataclasses were introduced first in Python 3.7 and allow the user to quickly put together simple data container classes using type hints. It is important to mention that the types are not enforced or that automatic conversions will be applied. This is still up to the user.\n",
        "\n",
        "The methods that are generated out of the box can be customised by specifying flags in the decorator e.g. by specifying `slots=True` the dataclass implementation will use slots for its fields instead of an internal `dict` (see \"Object-oriented Programming -> Slots\").\n",
        "\n",
        "Regarding performance and memory requirements simple `dict` operations will still outperform dataclasses but they _can_ be as fast as a class using slots. Compared to `namedtuple` the preference is very clear: use `dataclass` unless you have a specific reason to use a `namedtuple`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "54d72460",
      "metadata": {
        "id": "54d72460"
      },
      "outputs": [],
      "source": [
        "from dataclasses import dataclass\n",
        "from typing import Any\n",
        "\n",
        "@dataclass(repr=False, eq=False, order=False, slots=True)\n",
        "class DataclassDemo:\n",
        "    x: int\n",
        "    y: int\n",
        "    value: Any\n",
        "\n",
        "data = DataclassDemo(10, 12, dict())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a1a951b0",
      "metadata": {
        "tags": [],
        "id": "a1a951b0"
      },
      "source": [
        "#### [Defaultdict](https://docs.python.org/3/library/collections.html#defaultdict-objects)\n",
        "\n",
        "`defaultdict` is a subclass of `dict` and behaves just like regular dictionaries. The main difference is that you can provide a factory function for values. Whenever a key is not yet present in the dictionary a new entry is created using the default factory function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5a24f97c",
      "metadata": {
        "id": "5a24f97c"
      },
      "outputs": [],
      "source": [
        "from collections import defaultdict\n",
        "\n",
        "# provide a default value of '100' for all values\n",
        "a_dict = defaultdict(lambda: 100)\n",
        "print('defaultdict:', a_dict.keys())\n",
        "\n",
        "a_dict['a'] += 10\n",
        "print('add entry:', a_dict.keys())\n",
        "print('entry value:', a_dict['a'])\n",
        "\n",
        "a_dict['b'] -= 40\n",
        "print('add entry:', a_dict.keys())\n",
        "print('entry value:', a_dict['b'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3a63cc23-0d88-4e78-bd72-deef11cee226",
      "metadata": {
        "id": "3a63cc23-0d88-4e78-bd72-deef11cee226"
      },
      "source": [
        "This default factory function parameter enables us to simplify workflows such as this often seen construct:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a8ce9718-df02-4823-a212-57033a2cefb3",
      "metadata": {
        "tags": [],
        "id": "a8ce9718-df02-4823-a212-57033a2cefb3"
      },
      "outputs": [],
      "source": [
        "input_data = [('foo', 'bar'), ('foo', 'baz'), ('bar', 'boo')]\n",
        "values_per_key = {}\n",
        "for key, value in input_data:\n",
        "    if key not in values_per_key:\n",
        "        value_list = []\n",
        "        value_list.append(value)\n",
        "        values_per_key[key] = value_list\n",
        "    else:\n",
        "        values_per_key[key].append(value)\n",
        "\n",
        "print(values_per_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "dec44688-26e0-4159-bdd8-f6fde2d10825",
      "metadata": {
        "id": "dec44688-26e0-4159-bdd8-f6fde2d10825"
      },
      "source": [
        "Using 'defaultdict' it is possible to simply write:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "be860116-3f9b-4175-a780-90c56449a0cc",
      "metadata": {
        "tags": [],
        "id": "be860116-3f9b-4175-a780-90c56449a0cc"
      },
      "outputs": [],
      "source": [
        "from collections import defaultdict\n",
        "\n",
        "input_data = [('foo', 'bar'), ('foo', 'baz'), ('bar', 'boo')]\n",
        "values_per_key = defaultdict(list)\n",
        "for name, value in input_data:\n",
        "    values_per_key[name].append(value)\n",
        "print(values_per_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f90c1920",
      "metadata": {
        "id": "f90c1920"
      },
      "source": [
        "#### [Counter](https://docs.python.org/3/library/collections.html#counter-objects)\n",
        "\n",
        "A `Counter` class can be used to quickly count the occurrences of (hashable) items in a list. It is a subclass of `dict` and thus behaves like a dict were the keys are the items and the values are the counts. If a non-existing element is access the `Counter` instance will return `0` instead of raising a `KeyError`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "a1f80bc1",
      "metadata": {
        "id": "a1f80bc1"
      },
      "outputs": [],
      "source": [
        "from collections import Counter\n",
        "\n",
        "# feed in a string (=iterable) and count the occurences of each character\n",
        "cnt1 = Counter('Hello World!')\n",
        "print('cnt1: ', cnt1)\n",
        "\n",
        "# print out the most common character\n",
        "print(cnt1.most_common(1))\n",
        "\n",
        "# the constructor can also take in a mapping that already has counts\n",
        "cnt2 = Counter({'red': 3, 'blue': 1, 'green': 0})\n",
        "print('cnt2: ', cnt2)\n",
        "print(cnt2.most_common(1))\n",
        "print(cnt2.total())\n",
        "\n",
        "# it is also easily possible to combine the values of two counters\n",
        "cnt3 = Counter(('green', 'blue', 'yellow', 'blue'))\n",
        "print('cnt3: ', cnt3)\n",
        "\n",
        "cnt4 = cnt2 + cnt3\n",
        "print('cnt4: ', cnt4)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b02a167b",
      "metadata": {
        "id": "b02a167b"
      },
      "source": [
        "#### [ChainMap](https://docs.python.org/3/library/collections.html#chainmap-objects)\n",
        "\n",
        "A `ChainMap` can be used to virtually link multiple dictionaries together without discarding the original dicts. If a value gets updated it stores the updates in the first dict argument, lookups however search the full chain (from left to right)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "37727b75",
      "metadata": {
        "id": "37727b75"
      },
      "outputs": [],
      "source": [
        "from collections import ChainMap\n",
        "\n",
        "# create three different dictionaries for this example\n",
        "runtime_arguments = {}\n",
        "cmdline_arguments = {'b': False, 'c': 6.2830}\n",
        "default_arguments = {'a': 0, 'b': True, 'c': 3.1415}\n",
        "print('runtime_arguments:', runtime_arguments)\n",
        "print('cmdline_arguments:', cmdline_arguments)\n",
        "print('default_arguments:', default_arguments)\n",
        "\n",
        "# put them in a ChainMap (order matters)\n",
        "c_map = ChainMap(runtime_arguments, cmdline_arguments, default_arguments)\n",
        "print('ChainMap: ', c_map)\n",
        "\n",
        "# lookup keys and see how the value resolution works\n",
        "print('param a (from default_arguments):', c_map['a'])\n",
        "print('param b (from cmdline_arguments):', c_map['b'])\n",
        "print('param c (from cmdline_arguments):', c_map['c'])\n",
        "\n",
        "# adding an override via ChainMap for \"c\" sets the value in \"runtime_arguments\"\n",
        "c_map['c'] = 10.0\n",
        "print(c_map)\n",
        "\n",
        "print('runtime_arguments:', runtime_arguments)\n",
        "print('cmdline_arguments:', cmdline_arguments)\n",
        "print('default_arguments:', default_arguments)\n",
        "\n",
        "print('param c (now from runtime_arguments):', c_map['c'])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "91f4eb99",
      "metadata": {
        "id": "91f4eb99"
      },
      "source": [
        "#### UserDict, UserList, UserStr\n",
        "\n",
        "These classes exist to simplify user defined implementations of `dict`, `list` as well as `str`. They make their content available in the `data` attribute and thus allow operating on the data for e.g. custom implementations of existing functions or new functions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "e3c310a2",
      "metadata": {
        "id": "e3c310a2"
      },
      "outputs": [],
      "source": [
        "from collections import UserList\n",
        "\n",
        "class MyUserList(UserList):\n",
        "    def clear_odd(self):\n",
        "        \"\"\" Add a simple custom function to clear the add values (assumes that the items are int). \"\"\"\n",
        "        self.data = [v for v in self.data if v % 2 == 0]\n",
        "\n",
        "l = MyUserList([1, 2, 3, 4, 5, 6, 7, 8])\n",
        "print('Initial list:', l)\n",
        "l.clear_odd()\n",
        "print('After executing custom method: ', l)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "84a5c1d2",
      "metadata": {
        "tags": [],
        "id": "84a5c1d2"
      },
      "source": [
        "## Object-oriented Programming - Backgrounds\n",
        "\n",
        "Also see: __[Python Datamodel](https://docs.python.org/3/reference/datamodel.html)__\n",
        "\n",
        "In python everything is an object. Objects that are called classes are simply put just template objects that tell the python interpreter how to create an object of certain attributes & methods to operate on these attributes.\n",
        "\n",
        "The most simple way in python to define a class is `class C: pass`. This is equivalent to writing `C = type('C', (), dict())`. Both statements produce a new class object that can be considered a new `type`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "29e9cee1-ef04-422b-b4b3-8115c03a081e",
      "metadata": {
        "tags": [],
        "id": "29e9cee1-ef04-422b-b4b3-8115c03a081e"
      },
      "outputs": [],
      "source": [
        "class Test: pass\n",
        "print('class Test:', Test)\n",
        "\n",
        "Test = type('Test', (), dict())\n",
        "print('class Test:', Test)\n",
        "\n",
        "print('type of Test:', type(Test))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "b557c90e-0843-4fac-8271-94dc7e4bc2a1",
      "metadata": {
        "id": "b557c90e-0843-4fac-8271-94dc7e4bc2a1"
      },
      "source": [
        "### Creating new instances\n",
        "\n",
        "When a user wants to create a new `Test` instance by executing `Test()` the python interpreter first calls the `__new__` method on the class object. This method is by definition a class method and is responsible for creating a new object instance of the type (see [class methods](#Static-&-Class-attributes-and-methods)). Usually this is done by delegating to `type`'s `__new__` method (the base class). The purpose of `__new__` is to provide users with a hook, an extension mechanism to utilise the class creation process to customise certain aspects of the created object instance. In most cases it is not necessary to implement/use this method directly - see [Metaclasses / Metaprogramming](#Metaclasses-/-Metaprogramming) for other examples.\n",
        "\n",
        "After the object instance has been created the interpreter will then call the `__init__` method on the instance. This method needs to take care of internal initialisation of state and variables of the instance (aka a constructor).\n",
        "\n",
        "Once this process is completed the construction of a new object instance of type `Test` has been finished and the object can be used."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f0c1fa21-768f-4b53-9cbb-067c5a37a5a1",
      "metadata": {
        "tags": [],
        "id": "f0c1fa21-768f-4b53-9cbb-067c5a37a5a1"
      },
      "outputs": [],
      "source": [
        "class Test:\n",
        "    def __new__(cls, *args, **kwargs):\n",
        "        print('new instance: __new__')\n",
        "        return super().__new__(cls)\n",
        "\n",
        "    def __init__(self, *args, **kwargs):\n",
        "        super().__init__()\n",
        "        print('new instance: __init__')\n",
        "        self.test_attribute = 10\n",
        "\n",
        "test_instance = Test()\n",
        "print('instance:', test_instance)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "71dea14f",
      "metadata": {
        "id": "71dea14f"
      },
      "source": [
        "### Special attributes and methods"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "361bdcd9-56f7-4695-aa30-5686b92c7999",
      "metadata": {
        "id": "361bdcd9-56f7-4695-aa30-5686b92c7999"
      },
      "source": [
        "In the previous example we already had a first examples of special attributes and methods. In fact python uses them a lot to provide extension points and to introduce user defined behaviour.\n",
        "\n",
        "For example in a regular instance of a class all attributes that are created are stored in an internal dictionary. The field's name is `__dict__` and can be accessed and manipulated like any other python dict. The pattern `__<name>__` tells a user that this variable has a special meaning.\n",
        "\n",
        "A similar naming scheme is used for e.g. implementing a custom `str` conversion method (`__str__`), a custom `hash` implementation (`__hash__`) or for providing a custom 'official string representation' of the instance whenever `repr` is called (`__repr__`). If `__str__` is not implemented it falls back to the `__repr__` method to provide a string representation of the class.\n",
        "\n",
        "A python object can be inspected by passing it to the `dir` method. Since everything is an object in python `dir` can be equally used on `functions`, `lambdas` and all other python objects. The same is true for special attributes and methods: in fact every python object has and uses special attributes and methods."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8a6b52c7-fbe9-48c0-8e38-3ce008f787a9",
      "metadata": {
        "id": "8a6b52c7-fbe9-48c0-8e38-3ce008f787a9"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "class Test:\n",
        "    def __init__(self, attrib):\n",
        "        self.attrib = attrib\n",
        "\n",
        "    def __str__(self):\n",
        "        return f'Test instance with attrib={self.attrib}'\n",
        "\n",
        "    def __repr__(self):\n",
        "        return f'<Test attrib={self.attrib}>'\n",
        "\n",
        "t = Test(10)\n",
        "print('print instance:', t)\n",
        "\n",
        "print('internal dict:', t.__dict__)\n",
        "\n",
        "print(t.__str__)\n",
        "print('custom str:', str(t))\n",
        "\n",
        "print(t.__repr__)\n",
        "print('custom repr:', repr(t))\n",
        "\n",
        "print(t.__hash__)\n",
        "print('default hash:', hash(t), os.linesep)\n",
        "\n",
        "\n",
        "print('dir on t:', dir(t), os.linesep)\n",
        "\n",
        "print('dir on Test.__init__:', dir(Test.__init__))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "476c9beb-7424-4ea8-a9e3-7d6038553d75",
      "metadata": {
        "id": "476c9beb-7424-4ea8-a9e3-7d6038553d75"
      },
      "source": [
        "Other often used special methods are the comparison methods (`__lt__`, `__eq__`, `__gt__`, etc ...) and the attribute access methods (`__getattr__`, `__getattribute__`, `__setattr__`, ...).\n",
        "\n",
        "Since there are really lots of details to all of those function please refer to the python documentation for a complete list and description of all methods.\n",
        "\n",
        "An important realisation is that attributes, methods and most other characteristics of a class can thus be changed at runtime."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4e5985b2-4a6f-42b7-90e6-a86eca74baeb",
      "metadata": {
        "id": "4e5985b2-4a6f-42b7-90e6-a86eca74baeb"
      },
      "outputs": [],
      "source": [
        "class Test:\n",
        "    pass\n",
        "\n",
        "t = Test()\n",
        "\n",
        "# print class name\n",
        "print('class name: ', t.__class__.__name__)\n",
        "\n",
        "# dict is empty\n",
        "print('empty class instance: ', t.__dict__)\n",
        "\n",
        "# dynamically add a new attribute\n",
        "t.new_attribute = 'foo'\n",
        "\n",
        "# dict contains 'new_attribute' afterwards\n",
        "print('class instance afterwards: ', t.__dict__)\n",
        "\n",
        "# print default representation\n",
        "print('default repr of class instance: ', repr(t))\n",
        "\n",
        "# change __repr__ method at runtime for all instances\n",
        "Test.__repr__ = lambda self: f'<Test new_attribute={self.new_attribute}>'\n",
        "\n",
        "# print new representation\n",
        "print('new repr of class instance: ',repr(t))\n",
        "\n",
        "# dynamically replace constructor\n",
        "def new_init(self, other_attribute):\n",
        "    self.other_attribute = other_attribute\n",
        "\n",
        "Test.__init__ = new_init\n",
        "\n",
        "# create new instance using new constructor\n",
        "u = Test('other')\n",
        "\n",
        "# dict contains 'other_attribute'\n",
        "print('with new constructor: ', u.__dict__)\n",
        "\n",
        "try:\n",
        "    # creating a new instance now requires the new attribute\n",
        "    Test()\n",
        "except TypeError as e:\n",
        "    print('caught TypeError after setting new \"__init__\" method - ', e)\n",
        "\n",
        "# it is even possible (but discouraged) to change the class of an instance at runtime\n",
        "class BetterTest:\n",
        "    def foo(self):\n",
        "        return 'foo'\n",
        "\n",
        "t.__class__ = BetterTest\n",
        "print('after replacing type:', type(t))\n",
        "print('calling new method:', t.foo())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "63d4911c",
      "metadata": {
        "id": "63d4911c"
      },
      "source": [
        "### Slots\n",
        "\n",
        "If arbitrary attributes are not necessary a class can be restricted to only allow certain attributes by using so called slots. In addition to having a check in place that an accessed attribute is really existing it can also save memory and improve lookup speeds.\n",
        "\n",
        "Usually the attribute lookup uses the specified attribute name and traverses a number of possible hook-points (e.g. the internal `__dict__`) that could provide a value for the name.\n",
        "\n",
        "When using slots python will create descriptors for the specified attributes. This omits the creation of the internal `__dict__` attribute and lets the python interpreter omit some lookup steps and instead directly access the attributes by using these descriptors (see [Datamodel - Slots](https://docs.python.org/3/reference/datamodel.html#slots) for more information).\n",
        "\n",
        "Since there is no internal dict to store and retrieve attributes no new attributes can be added at runtime. Such an access fails with an `AttributeError` when slots are used.\n",
        "\n",
        "Note about `tooling.getsize`: The method `sys.getsizeof` doesn't take into account the actual size of the objects attributes - you only see the space that is required for the object instance itself. Since every attribute is stored in the `__dict__` attribute this won't give you the complete picture. In order to show the actual size difference when using slots, we are using a small snippet to recursively get the size of all things stored in the object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "33f1776f",
      "metadata": {
        "id": "33f1776f"
      },
      "outputs": [],
      "source": [
        "from sys import getsizeof\n",
        "from tooling.getsize import getsize\n",
        "\n",
        "class ClassExample:\n",
        "    def __init__(self, x, y, z):\n",
        "        self.x = x\n",
        "        self.y = y\n",
        "        self.z = z\n",
        "\n",
        "class_example = ClassExample(1, 2, 3)\n",
        "print('size of class layout with __dict__: ', getsizeof(class_example))\n",
        "print('total size of class with __dict__: ', getsize(class_example))\n",
        "\n",
        "class SlotExample:\n",
        "    __slots__ = ('x', 'y', 'z')\n",
        "\n",
        "    def __init__(self, x, y, z):\n",
        "        self.x = x\n",
        "        self.y = y\n",
        "        self.z = z\n",
        "\n",
        "slot_example = SlotExample(1, 2, 3)\n",
        "print('size of class with __slots__: ', getsizeof(slot_example))\n",
        "print('total size of class with __dict__: ', getsize(slot_example))\n",
        "print('__dict__ of class: ', getattr(slot_example, '__dict__', None))\n",
        "print('descriptor of SlotExample.x: ', SlotExample.x)\n",
        "\n",
        "try:\n",
        "    slot_example.new_attribute = 'bar'\n",
        "except AttributeError:\n",
        "    print('Caught AttributeError - assignment of new attributes is not possible when slots are used.')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4bc4b4b9",
      "metadata": {
        "id": "4bc4b4b9"
      },
      "source": [
        "### Inheritance\n",
        "\n",
        "Python also supports polymorphism and (multiple) inheritance. The inheritance hierarchy is defined by the MRO (method resolution order) and can be looked up at runtime using the `mro` method.\n",
        "\n",
        "Regarding method resolution order consider the following example of multiple inheritance (diamond problem) for demonstration purposes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1ed1d2fe",
      "metadata": {
        "id": "1ed1d2fe"
      },
      "outputs": [],
      "source": [
        "import abc      # see https://docs.python.org/3/library/abc.html\n",
        "\n",
        "class Animal(abc.ABC):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.speed = 1\n",
        "        self.smell = 1\n",
        "\n",
        "    def speak(self):\n",
        "        # default implementation raises a not implemented error\n",
        "        # note: raising an error prevents cooperative inheritance from working\n",
        "        raise NotImplementedError\n",
        "\n",
        "    def cuddle(self):\n",
        "        # default implementation does nothing\n",
        "        return ''\n",
        "\n",
        "    @abc.abstractmethod\n",
        "    def attack(self):\n",
        "        # default implementation does nothing\n",
        "        return ''\n",
        "\n",
        "class Dog(Animal):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.speed = 20\n",
        "        self.smell = 100\n",
        "\n",
        "    def speak(self):\n",
        "        return 'bark'\n",
        "\n",
        "    def cuddle(self):\n",
        "        return f'wince {super().cuddle()}'\n",
        "\n",
        "    def attack(self):\n",
        "        return f'grrrrr'\n",
        "\n",
        "class Cat(Animal):\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.speed = 10\n",
        "        self.smell = 20\n",
        "        self.purr_factor = 9000\n",
        "\n",
        "    def speak(self):\n",
        "        return 'meow'\n",
        "\n",
        "    def cuddle(self):\n",
        "        return f'purr {super().cuddle()}'\n",
        "\n",
        "    def attack(self):\n",
        "        return f'hiss'\n",
        "\n",
        "class DogCat(Dog, Cat):\n",
        "    pass\n",
        "\n",
        "class CatDog(Cat, Dog):\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "035945fe-1380-4a0a-be28-920c90d87e73",
      "metadata": {
        "id": "035945fe-1380-4a0a-be28-920c90d87e73"
      },
      "source": [
        "We are using `@abc.abstractmethod` on the method `attack`. In python this is a way to force all subclasses to implement a specific method. If a subclass does not implement the method it is itself considered an abstract class and cannot be instantiated.\n",
        "\n",
        "Note how the method `speak` which is also not implemented on `class Test` does not generate any error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ad3d3e28-4c4a-476f-bcdd-02ba4134d0a9",
      "metadata": {
        "id": "ad3d3e28-4c4a-476f-bcdd-02ba4134d0a9"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "    Animal()\n",
        "except TypeError as e:\n",
        "    print('Animal Error:', e)\n",
        "\n",
        "class Test1(Animal):\n",
        "    pass\n",
        "\n",
        "try:\n",
        "    Test1()\n",
        "except TypeError as e:\n",
        "    print('Test1 Error:', e)\n",
        "\n",
        "class Test2(Animal):\n",
        "    def attack(self):\n",
        "        return 'klonk'\n",
        "\n",
        "test2 = Test2()\n",
        "print('MRO of Test2: ', type(test2).mro())\n",
        "print(test2.attack())\n",
        "print('Internal dict of test2: ', test2.__dict__)\n",
        "\n",
        "print('is \"test2\" an instance of Animal: ', isinstance(test2, Animal))\n",
        "print('is \"test2\" an instance of Dog: ', isinstance(test2, Dog))\n",
        "print('is \"test2\" an instance of Cat: ', isinstance(test2, Cat))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "5ac689f2-0383-4094-8344-cc7048da11ee",
      "metadata": {
        "id": "5ac689f2-0383-4094-8344-cc7048da11ee"
      },
      "source": [
        "Next we're going to have a look at the classes `DogCat` and `CatDog`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "875a8165-32ce-4188-b3b1-e7da14627c59",
      "metadata": {
        "id": "875a8165-32ce-4188-b3b1-e7da14627c59"
      },
      "outputs": [],
      "source": [
        "kotpies = DogCat()\n",
        "print('MRO of DogCat: ', type(kotpies).mro())\n",
        "print('Internal dict of kotpies: ', kotpies.__dict__)\n",
        "\n",
        "# use 'isinstance' to check if an instance is of a given type\n",
        "print('is \"kotpies\" an instance of Animal: ', isinstance(kotpies, Animal))\n",
        "print('is \"kotpies\" an instance of Dog: ', isinstance(kotpies, Dog))\n",
        "print('is \"kotpies\" an instance of Cat: ', isinstance(kotpies, Cat))\n",
        "\n",
        "print('kotpies says:', kotpies.speak())\n",
        "print('cuddle kotpies:', kotpies.cuddle())\n",
        "print('kotpies attacks:', kotpies.attack())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "918c0c5e-5420-4136-834a-7b1316f7bb62",
      "metadata": {
        "id": "918c0c5e-5420-4136-834a-7b1316f7bb62"
      },
      "outputs": [],
      "source": [
        "pieskot = CatDog()\n",
        "print('MRO of CatDog: ', type(pieskot).mro())\n",
        "print('Internal dict of pieskot: ', pieskot.__dict__)\n",
        "\n",
        "# use 'isinstance' to check if an instance is of a given type\n",
        "print('is \"pieskot\" an instance of Animal: ', isinstance(pieskot, Animal))\n",
        "print('is \"pieskot\" an instance of Dog: ', isinstance(pieskot, Dog))\n",
        "print('is \"pieskot\" an instance of Cat: ', isinstance(pieskot, Cat))\n",
        "\n",
        "print('pieskot says:', pieskot.speak())\n",
        "print('cuddle pieskot:', pieskot.cuddle())\n",
        "print('pieskot attacks:', pieskot.attack())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d85a52c7",
      "metadata": {
        "id": "d85a52c7"
      },
      "source": [
        "From the above examples we can see that the MRO works by going from subclass(es) to baseclass until it stops at the implicit `object` base class every python type inherits from.\n",
        "\n",
        "Note that the order of the (parent-)classes in the definition of the subclasses matters. For example `DogCat` (`Dog` comes before `Cat`) vs `CatDog` (`Cat` comes before `Dog`).\n",
        "\n",
        "If you need cooperative inheritance to work for your constructors and methods, don't forget to call `super().<methodname>()` in the body of the specific method."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a1e30723-1be4-4d1b-bc79-37b4fb6f0e01",
      "metadata": {
        "id": "a1e30723-1be4-4d1b-bc79-37b4fb6f0e01"
      },
      "source": [
        "### Static & Class attributes and methods\n",
        "\n",
        "Classes can have so called static attributes, static methods and class methods.\n",
        "\n",
        "What this means is that the method is not considered to work on an instance of a class (`self`) but just resembles a free standing function. In the case of a class method the actual class object is given as the first argument.\n",
        "\n",
        "Since class objects are also just objects, they also have an internal `__dict__` and can store values as well. Those values are accessible directly via the class object itself and also via an instance. But be aware that direct assignment of a class attribute via an instance will actually create an attribute on the instance rather than changing the class attribute itself."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cd8946ff",
      "metadata": {
        "id": "cd8946ff"
      },
      "outputs": [],
      "source": [
        "class Static:\n",
        "    count = 10\n",
        "\n",
        "    @staticmethod\n",
        "    def static_method():\n",
        "        return f'static count is: \"{Static.count}\"'\n",
        "\n",
        "    @classmethod\n",
        "    def class_method(cls):\n",
        "        return f'class: \"{cls.__name__}\" bar - static count is \"{cls.count}\"'\n",
        "\n",
        "a, b = Static(), Static()\n",
        "print(a.count, b.count, Static.count)\n",
        "\n",
        "Static.count = 5\n",
        "print(a.count, b.count, Static.count)\n",
        "\n",
        "# be aware! when trying to access a static attribute like this, it creates a new instance attribute instead\n",
        "print(a.__dict__)\n",
        "a.count = 4\n",
        "print(a.__dict__)\n",
        "\n",
        "print(a.count, b.count, Static.count)\n",
        "\n",
        "# call the static method, it has access to static attributes\n",
        "print(Static.static_method())\n",
        "\n",
        "# call the class method, which is similar to static method but gets the type as first argument\n",
        "print(Static.class_method())\n",
        "\n",
        "# you can also call static and class methods via an instance\n",
        "print(a.static_method())\n",
        "print(b.class_method())"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a7459286",
      "metadata": {
        "id": "a7459286"
      },
      "source": [
        "### Metaclasses / Metaprogramming\n",
        "\n",
        "Since even types themselves are objects we can use that to our advantage. Creating a class derived from `type` can be used to add additional behaviour to instances created a this class. A class that creates other class instances in that manor is called a `metaclass`.\n",
        "\n",
        "This kind of mechanism is often used e.g. to realise ORM systems, to generate additional methods or code for definitions in classes.\n",
        "\n",
        "Note that every class in python can only have exactly one metaclass."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2ad0ccbc-3745-4b39-a6e6-3d862a3bc333",
      "metadata": {
        "id": "2ad0ccbc-3745-4b39-a6e6-3d862a3bc333"
      },
      "outputs": [],
      "source": [
        "from typing import Dict, List\n",
        "\n",
        "class MyMetaclass(type):\n",
        "    def __new__(cls, name, bases=None, namespace=None):\n",
        "        print('using \"MyMetaclass\"', name, bases, namespace)\n",
        "        instance = type.__new__(cls, name, bases, namespace)\n",
        "\n",
        "        fields = {key: value\n",
        "                  for key, value in namespace.items()\n",
        "                  if isinstance(value, MyField)}\n",
        "\n",
        "        for key, value in fields.items():\n",
        "            print(f'adding field: {key}; {value}')\n",
        "            setattr(instance, key, value.default)\n",
        "\n",
        "        def check(self):\n",
        "            for key, value in fields.items():\n",
        "                if not value.required:\n",
        "                    continue\n",
        "                if getattr(self, key) == None:\n",
        "                    raise ValueError(f'attribute must not be none: {key}')\n",
        "\n",
        "        # hook up the function\n",
        "        setattr(instance, 'check', check)\n",
        "\n",
        "        return instance\n",
        "\n",
        "# very basic field descriptor example\n",
        "class MyField:\n",
        "    def __init__(self, default=None, required=None):\n",
        "        self.default = default\n",
        "        self.required = required\n",
        "\n",
        "# just here to demonstrate where it shows up\n",
        "class BaseClass:\n",
        "    pass\n",
        "\n",
        "# subclass actually using the metaclass\n",
        "class SubClass(BaseClass, metaclass=MyMetaclass):\n",
        "    attrib1 = MyField(default=10, required=False)\n",
        "    attrib2 = MyField(required=True)\n",
        "\n",
        "# instantiate a new instance\n",
        "sc_a = SubClass()\n",
        "\n",
        "# print the values\n",
        "print(sc_a.attrib1)\n",
        "print(sc_a.attrib2)\n",
        "\n",
        "# run check\n",
        "try:\n",
        "    sc_a.check()\n",
        "except ValueError as e:\n",
        "    print('check failed:', e)\n",
        "\n",
        "# set attrib2 and try again\n",
        "sc_a.attrib2 = 23\n",
        "sc_a.check()\n",
        "print('check passed')"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    },
    "vscode": {
      "interpreter": {
        "hash": "7b25d2c9fa4339ca9c2d1c2c36fe6a22cc165ea5d14612a218506569477642ae"
      }
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}