John Swinbank added a comment - 05/Apr/16 4:46 AM Looking at HSC's ProcessExposureTask , I see four global tasks being performed after per-CCD processing: focus photometry astrometry curve of growth We know ( DM-2913 ) that the curve of growth code won't be ported to LSST. At the Monday morning meeting of 4 April, we discussed the photometric & astrometric solutions and agreed that they did not need to be provided by LSST to regard the HSC port as complete. I think, therefore, that no work on them is required here. Paul Price , do you agree? Per Paul Price 's clo posting of 4 April , "we can move [the focus solution] into its own job". Can you clarify exactly what this means? In order to close this issue, do we need to provide a new top-level task which provides distributed focus measurement?

Paul Price added a comment - 05/Apr/16 6:48 AM I will: 1. Move the focus measurement into its own script. We don't need to run it together with the science CCDs. Nate Lust can ignore the focus CCDs in his port. 2. Change the singleFrameDriver to operate over CCDs instead of exposures. This will let us parallelise much more widely and remove inefficiency from blocking on the exposure gather/scatter.

Paul Price added a comment - 29/Apr/16 11:44 AM Moving the focus measurement to its own script has been deferred to DM-5904 .

Paul Price added a comment - 29/Apr/16 11:46 AM Thanks for volunteering to review this, Simon Krughoff ! There is a single commit in each of ctrl_pool and pipe_drivers. pprice@tiger-sumire:/tigress/pprice/dm-5681/ctrl_pool (tickets/DM-5681=) $ git sub commit d00b4eb14b749a16b2a9748ed59f35f2c682c5ea Author: Paul Price <price@astro.princeton.edu> Date: Fri Apr 29 14:21:48 2016 -0400   add BatchParallelTask to provide simple iteration This new subclass of BatchCmdLineTask uses the MPI process pool as a mere iteration device, like a multi-node version of the '-j' command-line argument. Contrast this with BatchPoolTask, which allows the user to use the process pool directly (so they can do their own scatter/gather workflows).   python/lsst/ctrl/pool/parallel.py | 73 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 71 insertions(+), 2 deletions(-)     upprice@tiger-sumire:/tigress/pprice/dm-5681/pipe_drivers (tickets/DM-5681=) $ gt sub commit c2b27106ef3825ae907fe1de88761cf9ec2f1283 Author: Paul Price <price@astro.princeton.edu> Date: Fri Apr 29 14:24:44 2016 -0400   singleFrameDriver: convert to using BatchParallelTask We don't do any scatter/gather workflows, so all we really want is a multi-node version of the multiprocessing.Pool, which BatchParallelTask provides.   python/lsst/pipe/drivers/singleFrameDriver.py | 97 ++++----------------------- 1 file changed, 13 insertions(+), 84 deletions(-)

Simon Krughoff added a comment - 29/Apr/16 2:06 PM This looks fine. Just a couple comments. I don't have any way to test this.

Paul Price added a comment - 29/Apr/16 8:36 PM Thanks Simon Krughoff ! I added some more documentation (parameters, outputs of the function you identified), and justified my use of super on the github PR (it seems unavoidable). Then I gave it another test, and I'm happy to see that I can now parallelise wider than 112 cores in a single job. Merged to master.