Posts by author souchaud@phimeca

How to add a post

To add a post one must have a Trac account. If you do not have already one, ask one on users@… mailing list.

New distributed python wrapper


a new python wrapper is available on the souchaud branch : svn co (version 2691 is ok)

characteristic of the new wrapper :

  • ease-of-use: use python language and propose a module (coupling_tools) that helps manipulate templates
  • transparent compute distribution on each core of the local machine.
  • transparent compute distribution on each core of several hosts (needs a ssh server on each remote hosts, not yet implemented on windows).
  • transparent compute distribution using a remote cluster (not yet implemented)

An example is available in the branch: wrappers/WrapperTemplates/wrapper_python_distributed.
If you want to distribute computing onto several hosts, the python-paramiko module must be installed on the machine that launch the openturns script.

For further info on the DistributedPythonFunction and CouplingTools interface :

This wrapper should be tested in order to give us enough feedback. It is still under development and some part of its code has to be improved/cleanup...

This wrapper overhead is around 0.05s per point (constant from 1 to 8 cores), so external code that last less than 0.1s will not be speed up as fast as theoricaly. (e.g. 1000 point to compute. each point last 0.1s -> total computing time using one core : 1000*(0.1+0.05) = 150s (would be 100s without overhead)). Tested on i7 2.3Ghz*8 w/oHT + SSD.
The current C wrapper (e.g. in wrappers/WrapperTemplates/wrapper_calling_shell_command) overhead is around 0.01s per point (dependent to the number of core used: 1core->overhead=0.007s, 8cores->overhead=0.02s).

If you've got a huge number of point that last less that 0.01s, the best is to implement _exec_sample function directly (using C wrapper or OpenTURNSPythonFunction). If needed, the new wrapper could in the future, permit to implement _exec_sample in order to reduce the overhead.

If you've got some ideas or remarks, do not hesitate to make a comment, but check first it is not already in the todo file (in wrappers/WrapperTemplates/wrapper_python_distributed/TODO.txt).


Mathieu Souchaud

Fourth OpenTURNS Users Day slides

OpenTURNS Users Day slides are available here .