a new python wrapper is available on the souchaud branch : svn co https://svn.openturns.org/branches/souchaud (version 2691 is ok)
characteristic of the new wrapper :
- ease-of-use: use python language and propose a module (coupling_tools) that helps manipulate templates
- transparent compute distribution on each core of the local machine.
- transparent compute distribution on each core of several hosts (needs a ssh server on each remote hosts, not yet implemented on windows).
- transparent compute distribution using a remote cluster (not yet implemented)
An example is available in the branch: wrappers/WrapperTemplates/wrapper_python_distributed.
If you want to distribute computing onto several hosts, the python-paramiko module must be installed on the machine that launch the openturns script.
For further info on the DistributedPythonFunction and CouplingTools interface :
- see http://trac.openturns.org/browser/branches/souchaud/python/src/distributed_wrapper.py and http://trac.openturns.org/browser/branches/souchaud/python/src/coupling_tools.py files
- or :
import openturns; help(openturns.distributed_wrapper.DistributedPythonFunction.__init__) import openturns.coupling_tools; help(openturns.coupling_tools)
This wrapper should be tested in order to give us enough feedback. It is still under development and some part of its code has to be improved/cleanup...
This wrapper overhead is around 0.05s per point (constant from 1 to 8 cores), so external code that last less than 0.1s will not be speed up as fast as theoricaly. (e.g. 1000 point to compute. each point last 0.1s -> total computing time using one core : 1000*(0.1+0.05) = 150s (would be 100s without overhead)). Tested on i7 2.3Ghz*8 w/oHT + SSD.
The current C wrapper (e.g. in wrappers/WrapperTemplates/wrapper_calling_shell_command) overhead is around 0.01s per point (dependent to the number of core used: 1core->overhead=0.007s, 8cores->overhead=0.02s).
If you've got a huge number of point that last less that 0.01s, the best is to implement _exec_sample function directly (using C wrapper or OpenTURNSPythonFunction). If needed, the new wrapper could in the future, permit to implement _exec_sample in order to reduce the overhead.
If you've got some ideas or remarks, do not hesitate to make a comment, but check first it is not already in the todo file (in wrappers/WrapperTemplates/wrapper_python_distributed/TODO.txt).
In a post on the user list, one asks for an easy way to wrap a scilab code into an OpenTURNS function. Here are two possibilities:
- using the generic wrapper
- using the Python wrapping of OpenTURNS function, in two flavours
There are other possibilities I didn't have time to illustrate.
Even with this simple function (the sum of two floating point values), the generic wrapper is faster than the python wrapper by 34% on a quad-core computer.