attachment:notes_granger.txt of msri07/notes

   1 10:30 a.m.
   2 Interactive Parallel Computing with Python and IPython
   3 Brian Granger
   4 
   5 Python
   6 -Freely available (BSD License)
   7 -Highly portable:  OSX, Windows, Linux, supercomputers
   8 -Can be used interactively (like Matlab, Mathematica, IDL)
   9 -Simple, expressive syntax readable by human beings
  10 -Supports OO, functional, generic and meta programming
  11 -Large community of scientific/HPC users
  12 -Powerful built-in data types and libraries
  13 	-Strings, lists, sets, dictionaries (hash tables)
  14 	-Networking, XML parsing, thrading, regular expressions...
  15 -Larger number of third party libraries for scientific computing
  16 -Easy to wrap existing C/C++/Fortran codes
  17 
  18 IPython:  Enhanced Interactive Python Shell
  19 -Freely available (BSD license)  http://ipython.scipy.org/
  20 -Goal: Provide an efficient environment for exploratory and interactive scientific computing
  21 -The de facto shell for scientific computing in Python (has become in past few years)
  22 -Available as a standard package on every major Linux distribution.  Downloaded over 27k times in 2006 alone
  23 -Interactive Shell for many other projects:
  24 	-Math (SAGE)
  25 	-Astronomy (PyRAF,CASA)
  26 	-Physics (Ganga, PyMAD)
  27 	-Biology (Pymerase)
  28 	-Web Frameworks
  29 
  30 Capabilities:
  31 -Input/output histories
  32 -interactive GUI control:  enables interactive plotting
  33 -Highly customizable: extensible syntax, error handling, ...
  34 -Interactive control system: magic commands
  35 -Dynamic Introspection of nearly everything (objects, help, filesystem, etc.)
  36 -Direc acess to filesystem and shell
  37 -Integrated debugger and profiler support
  38 -Easy to embed: give any program an interactive console with one line of code
  39 -Interactive Parallel/Distributed Computing...
  40 
  41 Traditional Parallel Computing: (at least in the realm of physics)
  42 Compiled Languages:
  43 -C/C++/Fortran are FAST for computers, SLOW for you
  44 -Everything is low-level, you get nothing for free
  45 	-Only primitive data types
  46 	-Few built-in libraries
  47 	-Manual memory management: bugs and more bugs
  48 	-With C/C++ you don't even get built-in high performance numerical arrays
  49 
  50 Message Passing Interface: MPI
  51 Pros-
  52 	-Robust, optimized, standardized, portable, common
  53 	-Existing parallel libraries (FFTW, ScaLAPACK, Trillinos, PETSc)
  54 	-Runs over Ethernet, Infiniband, Myrinet
  55 	-Great at moving data around fast!
  56 Cons-
  57 	-Trivial things are not trivial.  Lots of boilerplate code.
  58 	-Orthogonal to how scientists think and work
  59 	-Static: load balancing and fault tolerance are difficult to implement
  60 	-Emphasis on compiled languages
  61 	-Non-interactive and non-collaborative
  62 	-Doesn't play well with other tools: GUIs, plotting, visualization, web
  63 	-Labor intensive to learn and use properly
  64 
  65 Case study: Parallel Jobs at NERSC in 2006:
  66 NERSC = DOE supercomputing center at Lawrence Berkeley National Laboratory
  67 Seaborg = IBM SP RS/6000 with 6080 CPUs
  68 	-90% of jobs used less than 113 CPUs
  69 	-Only 0.26% of jobs used more than 2048 CPUs
  70 Jacquard = 712 CPU Operton system
  71 	-50% of jobs used fewwer than 15 CPUs
  72 	-Only 0.39% of jobs used more than 256 CPUs
  73 Yes, this isn't the entire story.  There are other issues involved, reasons people don't want to use lots of CPUs.
  74 
  75 Realities:
  76 -Developing highly parallel codes with these tools is extremely difficult and time consuming
  77 -When it comes to parallel, WE are often the bottleneck
  78 -We spend most of our time writing code rather than waiting for those "slow" computers
  79 -With the advent of multi-core CPUs, this problem i scoming to a laptop/desktop near you
  80 -Parallel speedups are not guaranteed!
  81 
  82 Our Goals with IPython:
  83 -Trivial parallel things should be trivial
  84 -Difficult parallel things should be possible
  85 -Make all stages of parallel computing fully interactive: development, debugging, testing, execution, monitoring, ...
  86 -Make parallel computing more collaborative
  87 -More dynamic model for load balancing and fualt tolerance.
  88 -Seamless integration with other tools: plotting/visualization, system shell
  89 -Also want to keep the benefits of traditional approaches:
  90 	-Should be able to use MPI if it is appropriate
  91 	-Should be easy to integrate compiled code and libraries.
  92 -Support many types of parallelism.
  93 
  94 
  95 Computing with Namespaces:
  96 Namespaces:
  97 -Namespace = a container for objects and their unique identifiers
  98 	Very common with Python.  Almost a hash table
  99 -An instruction stream causes a namespace to evolve with time
 100 -Interactive computing: the instruction stream has a human agent as its runtime source at some level
 101 	Human is at the command at some point
 102 -A (namespace, instruction stream) is a higher level abstraction than a process or a thread.
 103 -Data in a namespace can be reated inplace (by instructions) or by external I/O (disk, network).
 104 
 105 Important Points:
 106 -Requirements for Interactive Computation:
 107 	-Alice/Bob must be able to send instruction stream to a namespace
 108 	-Alice/Bob must be able to push/pull objects to/from the namespace (disk, network)
 109 -Requirements for Parallel Computation:
 110 	-Multiple namespaces and instruction streams (for general MIMD parallelism).
 111 	-Send data between namespaces (MPI is REALLY good at this).
 112 -Requirements for Interactive Parallel Computation:
 113 	-Alice/Bob must be able to send multiple instruction streams to multiple namespaces
 114 	-User must be able to push/pull objects to/from the namespaces
 115 THESE REQUIREMENTS HOLD for any type of parallelism
 116 
 117 IPython Architecture:
 118 pic
 119 
 120 Architecture Details:
 121 The IPython Engine/Controller/Client are typically different processes.  Why not threads?  Later
 122 -Can be run in arbitrary configurations on laptops, clusters, supercomputers
 123 -Everything is asynchronous.  Can't hack this on as an afterthought
 124 -Must deal with long running commands that block all network traffic
 125 -Dynamic process model.  Engines and Clients can come and go at will at any time (unless you're using MPI)
 126 
 127 
 128 Mapping Namespaces to Various Models of Parallel Computation:
 129 Key Points:
 130 -Most models of parallel/distributed computing can be mapped onto this architecture.
 131 	-Message passing
 132 	-Task farming
 133 	-TupleSpaces
 134 	-BSP (Bulk Synchronous Parallel)
 135 	-Google's MapReduce
 136 	-???
 137 -With IPython's architecture of all these types of parallel computations can be done Interactively and collaboratively.
 138 -The mapping of these models onto our architecture is done using interfaces + adapters and requires very little code.
 139 
 140 
 141 Live Demo:
 142 easy execution of commands on machines, easy to see what machines are connected, output of responses from machines as they
 143 come in.  One way they abstracte the asynchronicity is to use block=false to just return after it submits the command
 144 Magic commands are simply handled by such example:  %px import numpy
 145 %px  		parallel execute
 146 %pn #		execute only on machine #
 147 %result		get results of last things
 148 %autopx		automatically send everything sent to all machines
 149 rc.block=BOOL	show blocking or not
 150 rc.pushall	Send data
 151 rc.pullall	Get data
 152 Can use dictionary syntax:  rc.pushAll(d=3564567) = rc['d'] = 3564567
 153 rc.scatterAll	Scatters an array across machines
 154 rc.gatherAll	Gathers the array back
 155 The parallel time on the example rc.mapAll('lambda x: x**10', range(1000000)) is horrible
 156 There's lots of tools here, but you still have to think.
 157 
 158 Task-Based Computing:
 159 -Common style of parallelism for loosely coupled or independent tasks
 160 
 161 Stepping Back a Bit:
 162 -Event based systems are a nice alternative to threads:
 163 	-Scale to multiple CPU systems
 164 	-Bu9ild asynchronous nature of things in at low level
 165 	-No deadlocks to worry about
 166 -The networking framework used by IPython and SAGE (Twisted) has an abstraction (called a Deferred) for a result that will
 167 arrive at some point in the future (like a promise in E)
 168 -We have n interface that uses Deferreds to encapsulate asynchronous results/errors.
 169 
 170 Benefits of an Event Based System:
 171 -Arbitrary configurations of namespaces are immediately possible without worrying about deadlocks
 172 	-Our test suite create a controller/client and multiple engines in a single process
 173 -Possibiliities:
 174 	-Hierarchies of client/contro...
 175 	-Recursive systems
 176 
 177 Error Propagation:
 178 -Building a distributed system is easy...
 179 -Unless you want to handle errors well
 180 -We have spent a lot of time working/thinking about this issue.  Not always obvious what should happen.
 181 -Our goal:  error handling/propagation in a parallel distributed context should be a nice analytic continuation of what
 182 happens in a serial context.
 183 -Remote exceptions should propagate to the user in a meaningful manner
 184 -Policy of safety: don't ever let errors pass silently
 185 
 186 This week:
 187 -All the core developers of IPython's parallel capabilities are here at the workshop.