High performance computing for non-programmers

Glib Ivashkevych
junior researcher, NSC KIPT

Lecture #4: Python programming language: Advanced

List comprehensions rationale

newlist = [] for elem in oldlist: newlist.append(somefunc(elem))

Very common pattern Special syntax for this?

List comprehensions syntax

newlist = [] for elem in oldlist: newlist.append(somefunc(elem))

is equivalent to:
newlist = [somefunc(elem) for elem in oldlist]

List comprehensions syntax

zlist = [] for x in xlist: for y in ylist: zlist.append(func(x, y))

is equivalent to:
zlist = [func(x,y) for x in xlist for y in ylist]

List comprehensions syntax

zlist = [] for x in xlist: for y in ylist: if cond(x,y): zlist.append(f(x, y))

is equivalent to:
zlist = [f(x,y) for x in xlist for y in ylist if cond(x,y)]

Generators rationale
for elem in somelist: do_something(elem)

What if somelist is huge? Large memory footprint for storing somelist

Generators yield instruction

def huge_generator(num): for i in xrange(num): yield someexpr(i) some_generator = huge_generator(huge_number) for elem in some_generator: do_something(elem)

A lot better now. Generators know how to get next element (some_generator.next())

Generators expression syntax

def huge_generator(num): for i in xrange(num): yield f(i) sgen = huge_generator(huge_number)

is equivalent to:
sgen = (f(i) for i in xrange(huge_number))

Iterators behind the scenes

for elem in somelist: ... for elem in sometuple: ... for elem in someiterable: ...

Some pattern exists here, yeap?

Iterators behind the scenes

for calls __iter()__ on container object __iter()__ returns iterator object iterator knows how to get next element and where to stop
(raises StopIteration exception, when there are no more elements)

Iterators how to create

class CustomIterable(object): def __init__(self, *args, **kwargs): #initialize your object here def __iter__(self): return self def next(self): #return some value or raise StopIteration

CustomIterable can now be used in loops

Magic methods what about this __ methods?

Reserved methods with special meaning
for el in lst:

uses
LstClass.__iter__(lst) obj = SomeClass(args)

uses
SomeClass.__new__(SomeClass, args) SomeClass.__init__(obj, args)

and

Magic methods dirty secrets of constructors

__init__ is not a constructor (its initializer) although its usually to treat it as such except when subclassing immutables __new__ is an actual constructor you need it when subclassing immutables

Magic methods be polite

__str__ is used to emulate str() built-in function __unicode__ is used to emulate unicode() built-in function __repr__ is used to get a string representation of object (also `obj` and repr() built-in)

Magic methods be a container

__len__ is used to emulate len()
len(cont) -> cont.__len__() or ContClass.__len__(cont)

__getitem__ is used to emulate obj[key]

obj[key] -> obj.__getitem__(key) or ObjClass.__getitem__(obj, key)

Magic methods be a container

__setitem__ is used to emulate obj[key] =value
obj[key] = value -> obj.__setitem__(key, value) or ObjClass.__getitem__(obj, key, value)

Magic methods be a container

__delitem__(self, key) : del obj[key] __iter__(self) : iter(obj) or in loops __reversed__(self) : reversed(obj) __contains__(self, item) : item in obj

Magic methods be smart

__setattr__(self, name, value) Intercept assignment to attribute name Be aware of infinite recursion! __getattribute__(self, name) __getattr__(self, name) Intercept access to attribute name

Magic methods be callable

__call__(self, ...) Intercept call on instance class SomeClass(object): def __call__(self, *args, **kwargs): do_something s = SomeClass() s(1,2,3)

Magic methods all together

Magic methods in Python allow you to create classes, which look and behave like built-in types

See this blog post for general overview

http://www.rafekettler.com/magicmethods.html

Decorators rationale
Add functionality without changing the code

Examples: logging, timing, tests

Decorators rationale
def wrapper(f): def inner(*args): print Gonna call %s % f.func_name f(*args) return inner def func(*args): do_something func = wrapper(func) #additional functionality

Decorators rationale
def wrapper(f): def inner(*args): print Gonna call %s % f.func_name f(*args) return inner @wrapper def func(*args): do_something

Decorators rationale
And now magic happens: wrapper can be used to decorate any function. Code reuse, good design - PROFIT! ... @wrapper def f1(*args): do_something @wrapper def f2(*args): do_something_else ...

Decorators parametrized
What if your decorator depends on some value? Thats ok.
def dec(a): def wrapper(f): This is actual decorator def inner(*args): print Func is %s, param is %s % (f.func_name,str(a)) f(*args) return inner return wrapper @dec(Some string parameter) def func(*args): ...

Decorators leveraging __call__

class MyDecorator(object): def __init__(self, f): self.f = f def __call__(self, *args, **kwargs): print Starting %s call %s self.f.func_name self.f(*args, **kwargs) print Finished %s %s self.f.func_name @MyDecorator def func(*args, **kwargs): ...

Cool. Really.

Metaprogramming code that creates code

class MyClass(object): def __init__(self, data): self.data = data Something interesting ... ...

Classes and instances can be extended at runtime

Metaprogramming code that creates code

>>> MyClass.new_method = lambda self: self.data * 2 >>> m = MyClass(5.) >>> m.new_method() 10.

Classes and instances can be extended at runtime

Metaprogramming rationale
class IsothermalIdealGas(object): pressure = 1. temp = 1. volume = 1. quant = 1.

Not cool. Really. PV = nT? Never mind.

(forget about R, this is computer science class)

Developer of IsothermalIdealGas never heard about equation of state. Lets patch it.

Metaprogramming templating: jinja2

from jinja2 import Template tpl = Template(def custom_setattr(self, name, value): if name == 'P': object.__setattr__(self, name, value) object.__setattr__(self, 'V', {{V_eq}}) elif name == 'V': object.__setattr__(self, name, value) object.__setattr__(self, 'P', {{P_eq}})) v_eq = Template(self.{{T}}*self.{{Q}} / self.{{V}}**{{k}}) p_eq = Template((self.{{T}}*self.{{Q}} / self.{{P}})**(1./ {{k}})) keys = {T:T, P:P, V:V, Q:Q, k:2} source = tpl.render(V_eq=v_eq.render(...), P_eq=p_eq.render(...))

Metaprogramming templating: jinja2

Template variables
{{var}}, {{var.someattr}}, {{var[key]}}

for block
{% for item in iterable %} ... {% endfor %}

if block
{% if var%} ... {% endif %}

Metaprogramming templating: jinja2

from jinja2 import Template tpl = Template({{foo}} - {{bar}}) rtpl = tpl.render(foo=First,bar=Second) or rtpl = tpl.render({foo:First,bar:Second})

Ok, now we can generate code at runtime. Usually, for C/CUDA code.

Metaprogramming warning Never, evertrust other peoples data when using it to generate executable code. Always assume the worst. They all want to crash your code. Always.
(google it, yeah. something like python eval dangerous)

Python
With great power comes great responsibility.
- Uncle Ben from SpiderMan or, for those who prefer classic:

Un grand pouvoir implique de grandes responsabilits.

- Franois-Marie Arouet aka Voltaire

Questions?