We should consider delaying this until DM-4255 is complete, as that's a complete rewrite of ProcessCcdTask, and the right solution here may just be to turn ProcessCcdTask into a BatchPoolTask rather than putting another layer around it.

We discussed this at sprint planning of 2016-03-31. We agreed that the stopping condition for this ticket is to provide a layer capable of coordinating multiple executions of processCcd (covering the entire exposure), not to perform "global" processing as is currently carried out on HSC.

This basically looks fine, though I requested some cleanups on github

Russell Owen I think I have address everything significant. I am going to ask Dominique Boutigny if he would look at the new obs_cfht branch and Hsin-Fang Chiang if she would look at the obs_decam branch. Kian-Tat Lim In this ticket I only plan on adding support to run the single frame driver task to HSC, DECAM, and CFHT. The LSST telescopes have a different dataId structure that will need to be adapted in the future, and SDSS does not really fit into the visit, ccd model of things. This does not preclude support being added in the future, but it a bit outside of this porting ticket. Do you have any objections?

I confirm that singleFrameDriver runs with raw DECam data, with the two minor corrections noted in the obs_decam PR.

I do have another request (or set of related requests), though you can file a ticket and do it later if you need to:

• add a doc directory with the usual content, so the doc strings get generated by Doxygen
• Add a main.dox overview file to the doc directory so users have some idea what this package is for and how to use it
• Make sure all doc strings start with """! do Doxygen parses the doxygen commands

Aside from that it looks fine to me.

I tried it with obs_cfht. It runs but never ends ! It is looping over the CCDs indefinitely

This is a long shot, Dominique Boutigny would you mind trying branch u/hfc/DM-3368? I am guessing obs_cfht may need a fix that was previously done to obs_decam.

Hsin-Fang ChiangDominique Boutigny I fixed those issues before Dominique started looking at this. I am seeing the behavior on lsst-dev as well and am looking into it. I think there might be additional cfht specific code that might need to go in.

I suspected a possible bug equivalent to DM-5586 for obs_cfht (a MegacamMapper bug, not a pipe_drivers bug). If I understand correctly, with that bug, it would loop (the number of extension)*(the number of state) times more than it should. It seems to me that Megacam data can have 36 extensions and 2 states, so at most 36*2 = 72 times.

If it goes on indefinitely (more than 72 times), please disregard my random guess....

It is absolutely the same kind of problem as DM-5586.

Before:

>>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/')

>>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587}))

>>> len(refList)

36

Fix:

price@lsst-dev:~/LSST/obs/cfht (master *) $ git diff

diff --git a/policy/MegacamMapper.paf b/policy/MegacamMapper.paf

index 25a1e3a..ded990a 100644

--- a/policy/MegacamMapper.paf

+++ b/policy/MegacamMapper.paf

@@ -14,7 +14,7 @@ levels: {

     # Keys that are NOT relevant for a particular level

     skyTile: "visit" "ccd"

     tract: "patch"

-    visit: "ccd"

+    visit: "ccd" "extension" "state"

     sensor: "none"

 }

 defaultLevel: "Ccd"

After:

>>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/')

>>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587}))

>>> len(refList)

1

Oh, my fix is exactly the same as Hsin-Fang Chiang's.

These problems are coming up because no-one has cared about getting visit-level data refs for these cameras before now.

Thanks all for catching that all, and I appologize about not getting what you meant at first hsin-fang, I'm very foggy today. I am lucky to have other team members who are great. I will put these fixes on my branch first thing Monday.

I added the fix to my obs_cfht branch, and everything now runs to completion. Dominique Boutigny can you verify? And if so do you have any other review comments?

I confirm that this is now running and ending... with the latest mods in obs_cfht.
Just one question :

• How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?

You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.

Dominique Boutigny, put in your obs package's config/singleFrameDriver.py:

import os.path

from lsst.utils import getPackageDir

config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py"))

This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd.

If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top:

try:

    config = config.processCcd

except:

    pass

With this, you can use the overrides with either the singleFrameDriver or processCcd.

Paul Price, The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.

merged to master