# Port HSC MPI driver for single-visit processing

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
6
• Sprint:
Science Pipelines DM-W16-6, DRP X16-1, DRP X16-2
• Team:
Data Release Production

## Description

Transfer the reduceFrames.py script and the ProcessExposureTask it utilizes from hscPipe to a new package in the LSST stack (RFC-68 proposes calling this new package pool_tasks, but this isn't set in stone).

We should probably rename either the driver script or the task (or both), so they agree; the lack of consistency is a historical artifact on the HSC side, and I think it's time to change that.

## Activity

Jim Bosch created issue -
Field Original Value New Value
Epic Link DM-1942 [ 16008 ]
 Link This issue is blocked by DM-2983 [ DM-2983 ]
 Rank Ranked higher
 Story Points 2 4
 Epic Link DM-1942 [ 16008 ] DM-3560 [ 19650 ]
 Epic Link DM-3560 [ 19650 ] DM-3568 [ 19662 ]
 Assignee Nate Lust [ nlust ]
 Sprint Science Pipelines DM-W16-1 [ 173 ]
 Rank Ranked higher
 Sprint Science Pipelines DM-W16-1 [ 173 ] Science Pipelines DM-W16-1, Science Pipelines DM-W16-2 [ 173, 176 ]
 Rank Ranked higher
 Sprint Science Pipelines DM-W16-1, Science Pipelines DM-W16-2 [ 173, 176 ] Science Pipelines DM-W16-1, Science Pipelines DM-W16-3 [ 173, 182 ]
 Sprint Science Pipelines DM-W16-1, Science Pipelines DM-W16-3 [ 173, 182 ] Science Pipelines DM-W16-1, Science Pipelines DM-W16-3, Science Pipelines DM-W16-4 [ 173, 182, 187 ]
 Rank Ranked higher
Hide
Jim Bosch added a comment -

We should consider delaying this until DM-4255 is complete, as that's a complete rewrite of ProcessCcdTask, and the right solution here may just be to turn ProcessCcdTask into a BatchPoolTask rather than putting another layer around it.

Show
Jim Bosch added a comment - We should consider delaying this until DM-4255 is complete, as that's a complete rewrite of ProcessCcdTask , and the right solution here may just be to turn ProcessCcdTask into a BatchPoolTask rather than putting another layer around it.
 Link This issue is blocked by DM-4255 [ DM-4255 ]
 Sprint Science Pipelines DM-W16-1, Science Pipelines DM-W16-3, Science Pipelines DM-W16-4 [ 173, 182, 187 ] Science Pipelines DM-W16-1, Science Pipelines DM-W16-3, Science Pipelines DM-W16-4, Science Pipelines DM-W16-5 [ 173, 182, 187, 193 ]
 Rank Ranked higher
 Link This issue is contained by DM-4855 [ DM-4855 ]
 Status To Do [ 10001 ] In Progress [ 3 ]
 Rank Ranked higher
 Sprint Science Pipelines DM-W16-1, Science Pipelines DM-W16-3, Science Pipelines DM-W16-4, Science Pipelines DM-W16-5 [ 173, 182, 187, 193 ] Science Pipelines DM-W16-6 [ 195 ]
 Component/s pipe_tasks [ 10726 ] Component/s hscPipe [ 10736 ]
 Sprint Science Pipelines DM-W16-6 [ 195 ] Science Pipelines DM-W16-6, Alert Production X16 [ 195, 203 ]
 Rank Ranked higher
 Sprint Science Pipelines DM-W16-6, Alert Production X16 [ 195, 203 ] Science Pipelines DM-W16-6, DRP W16-7 [ 195, 208 ]
 Rank Ranked lower
 Link This issue is contained by DM-4855 [ DM-4855 ]
 Epic Link DM-3568 [ 19662 ] DM-5399 [ 23206 ]
 Sprint Science Pipelines DM-W16-6, DRP W16-7 [ 195, 208 ] Science Pipelines DM-W16-6 [ 195 ]
 Sprint Science Pipelines DM-W16-6 [ 195 ] Science Pipelines DM-W16-6, DRP X16-1 [ 195, 210 ]
 Story Points 4 6
 Link This issue is blocked by DM-5284 [ DM-5284 ]
Hide
John Swinbank added a comment -

We discussed this at sprint planning of 2016-03-31. We agreed that the stopping condition for this ticket is to provide a layer capable of coordinating multiple executions of processCcd (covering the entire exposure), not to perform "global" processing as is currently carried out on HSC.

Show
John Swinbank added a comment - We discussed this at sprint planning of 2016-03-31. We agreed that the stopping condition for this ticket is to provide a layer capable of coordinating multiple executions of processCcd (covering the entire exposure), not to perform "global" processing as is currently carried out on HSC.
 Sprint Science Pipelines DM-W16-6, DRP X16-1 [ 195, 210 ] Science Pipelines DM-W16-6, DRP X16-1, DRP X16-2 [ 195, 210, 214 ]
 Rank Ranked higher
 Reviewers Russell Owen [ rowen ] Status In Progress [ 3 ] In Review [ 10004 ]
Hide
Russell Owen added a comment -

This basically looks fine, though I requested some cleanups on github

Show
Russell Owen added a comment - This basically looks fine, though I requested some cleanups on github
 Status In Review [ 10004 ] Reviewed [ 10101 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
Hide
Nate Lust added a comment -

Russell Owen I think I have address everything significant. I am going to ask Dominique Boutigny if he would look at the new obs_cfht branch and Hsin-Fang Chiang if she would look at the obs_decam branch. Kian-Tat Lim In this ticket I only plan on adding support to run the single frame driver task to HSC, DECAM, and CFHT. The LSST telescopes have a different dataId structure that will need to be adapted in the future, and SDSS does not really fit into the visit, ccd model of things. This does not preclude support being added in the future, but it a bit outside of this porting ticket. Do you have any objections?

Show
Nate Lust added a comment - Russell Owen I think I have address everything significant. I am going to ask Dominique Boutigny if he would look at the new obs_cfht branch and Hsin-Fang Chiang if she would look at the obs_decam branch. Kian-Tat Lim In this ticket I only plan on adding support to run the single frame driver task to HSC, DECAM, and CFHT. The LSST telescopes have a different dataId structure that will need to be adapted in the future, and SDSS does not really fit into the visit, ccd model of things. This does not preclude support being added in the future, but it a bit outside of this porting ticket. Do you have any objections?
Hide
Hsin-Fang Chiang added a comment -

I confirm that singleFrameDriver runs with raw DECam data, with the two minor corrections noted in the obs_decam PR.

Show
Hsin-Fang Chiang added a comment - I confirm that singleFrameDriver runs with raw DECam data, with the two minor corrections noted in the obs_decam PR.
Hide
Russell Owen added a comment - - edited

I do have another request (or set of related requests), though you can file a ticket and do it later if you need to:

• add a doc directory with the usual content, so the doc strings get generated by Doxygen
• Add a main.dox overview file to the doc directory so users have some idea what this package is for and how to use it
• Make sure all doc strings start with """! do Doxygen parses the doxygen commands

Aside from that it looks fine to me.

Show
Russell Owen added a comment - - edited I do have another request (or set of related requests), though you can file a ticket and do it later if you need to: add a doc directory with the usual content, so the doc strings get generated by Doxygen Add a main.dox overview file to the doc directory so users have some idea what this package is for and how to use it Make sure all doc strings start with """! do Doxygen parses the doxygen commands Aside from that it looks fine to me.
Hide
Dominique Boutigny added a comment -

I tried it with obs_cfht. It runs but never ends ! It is looping over the CCDs indefinitely

Show
Dominique Boutigny added a comment - I tried it with obs_cfht. It runs but never ends ! It is looping over the CCDs indefinitely
 Link This issue relates to DM-5837 [ DM-5837 ]
Hide
Hsin-Fang Chiang added a comment -

This is a long shot, Dominique Boutigny would you mind trying branch u/hfc/DM-3368? I am guessing obs_cfht may need a fix that was previously done to obs_decam.

Show
Hsin-Fang Chiang added a comment - This is a long shot, Dominique Boutigny would you mind trying branch u/hfc/ DM-3368 ? I am guessing obs_cfht may need a fix that was previously done to obs_decam.
Hide
Nate Lust added a comment -

Hsin-Fang ChiangDominique Boutigny I fixed those issues before Dominique started looking at this. I am seeing the behavior on lsst-dev as well and am looking into it. I think there might be additional cfht specific code that might need to go in.

Show
Nate Lust added a comment - Hsin-Fang Chiang Dominique Boutigny I fixed those issues before Dominique started looking at this. I am seeing the behavior on lsst-dev as well and am looking into it. I think there might be additional cfht specific code that might need to go in.
Hide
Hsin-Fang Chiang added a comment -

I suspected a possible bug equivalent to DM-5586 for obs_cfht (a MegacamMapper bug, not a pipe_drivers bug). If I understand correctly, with that bug, it would loop (the number of extension)*(the number of state) times more than it should. It seems to me that Megacam data can have 36 extensions and 2 states, so at most 36*2 = 72 times.

If it goes on indefinitely (more than 72 times), please disregard my random guess....

Show
Hsin-Fang Chiang added a comment - I suspected a possible bug equivalent to DM-5586 for obs_cfht (a MegacamMapper bug, not a pipe_drivers bug). If I understand correctly, with that bug, it would loop (the number of extension)*(the number of state) times more than it should. It seems to me that Megacam data can have 36 extensions and 2 states, so at most 36*2 = 72 times. If it goes on indefinitely (more than 72 times), please disregard my random guess....
Hide
Paul Price added a comment -

It is absolutely the same kind of problem as DM-5586.

Before:

 >>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/') >>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587})) >>> len(refList) 36 

Fix:

 price@lsst-dev:~/LSST/obs/cfht (master *) $git diff diff --git a/policy/MegacamMapper.paf b/policy/MegacamMapper.paf index 25a1e3a..ded990a 100644 --- a/policy/MegacamMapper.paf +++ b/policy/MegacamMapper.paf @@ -14,7 +14,7 @@ levels: {  # Keys that are NOT relevant for a particular level  skyTile: "visit" "ccd"  tract: "patch" - visit: "ccd" + visit: "ccd" "extension" "state"  sensor: "none"  }  defaultLevel: "Ccd"  After:  >>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/') >>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587})) >>> len(refList) 1  Show Paul Price added a comment - It is absolutely the same kind of problem as DM-5586 . Before: >>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/') >>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587})) >>> len(refList) 36 Fix: price@lsst-dev:~/LSST/obs/cfht (master *)$ git diff diff --git a/policy/MegacamMapper.paf b/policy/MegacamMapper.paf index 25a1e3a..ded990a 100644 --- a/policy/MegacamMapper.paf +++ b/policy/MegacamMapper.paf @@ -14,7 +14,7 @@ levels: {      # Keys that are NOT relevant for a particular level      skyTile: "visit" "ccd"      tract: "patch" -    visit: "ccd" +    visit: "ccd" "extension" "state"      sensor: "none" } defaultLevel: "Ccd" After: >>> butler = Butler('/raid/nlust/repos/validation_data_cfht/data/') >>> refList = list(butler.subset(datasetType="raw", level="visit", dataId={'visit': 850587})) >>> len(refList) 1
Hide
Paul Price added a comment -

Oh, my fix is exactly the same as Hsin-Fang Chiang's.

These problems are coming up because no-one has cared about getting visit-level data refs for these cameras before now.

Show
Paul Price added a comment - Oh, my fix is exactly the same as Hsin-Fang Chiang 's. These problems are coming up because no-one has cared about getting visit-level data refs for these cameras before now.
Hide
Nate Lust added a comment -

Thanks all for catching that all, and I appologize about not getting what you meant at first hsin-fang, I'm very foggy today. I am lucky to have other team members who are great. I will put these fixes on my branch first thing Monday.

Show
Nate Lust added a comment - Thanks all for catching that all, and I appologize about not getting what you meant at first hsin-fang, I'm very foggy today. I am lucky to have other team members who are great. I will put these fixes on my branch first thing Monday.
Hide
Nate Lust added a comment -

I added the fix to my obs_cfht branch, and everything now runs to completion. Dominique Boutigny can you verify? And if so do you have any other review comments?

Show
Nate Lust added a comment - I added the fix to my obs_cfht branch, and everything now runs to completion. Dominique Boutigny can you verify? And if so do you have any other review comments?
Hide
Dominique Boutigny added a comment -

I confirm that this is now running and ending... with the latest mods in obs_cfht.
Just one question :

• How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?
Show
Dominique Boutigny added a comment - I confirm that this is now running and ending... with the latest mods in obs_cfht. Just one question : How can I pass a configuration parameter file as I do with processCcd with the --configfile parameter ?
Hide
Nate Lust added a comment -

You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.

Show
Nate Lust added a comment - You should be able to do it just the same way, just noting that processccd is a subtask (hence sub configuration) of SingleFrameDriver test. So your configuration file would be something like config.processCcd.<configuration variable> = <value> instead of just config.<configuration variable> = <value>. Everything will just be relative to the configuration of the task you are running.
Hide
Paul Price added a comment -

Dominique Boutigny, put in your obs package's config/singleFrameDriver.py:

 import os.path from lsst.utils import getPackageDir config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py")) 

This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd.

If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top:

 try:  config = config.processCcd except:  pass 

With this, you can use the overrides with either the singleFrameDriver or processCcd.

Show
Paul Price added a comment - Dominique Boutigny , put in your obs package's config/singleFrameDriver.py: import os.path from lsst.utils import getPackageDir config.processCcd.load(os.path.join(getPackageDir("obs_subaru"), "config", "processCcd.py")) This ensures you're using the same overrides in the singleFrameDriver as you use for processCcd. If you've got an override file that's not being automatically loaded from your obs package, you can put this at the top: try: config = config.processCcd except: pass With this, you can use the overrides with either the singleFrameDriver or processCcd.
Hide
Nate Lust added a comment -

Paul Price, The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.

Show
Nate Lust added a comment - Paul Price , The config file that I built for cfht already has that in it, so the only overrides Dominique Boutigny should need are run specific things if he wants to change an option between runs or for a specific purpose.
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
Hide
Nate Lust added a comment -

merged to master

Show
Nate Lust added a comment - merged to master
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Link This issue blocks DM-5681 [ DM-5681 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
Pim Schellart [X] (Inactive) made changes -
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]
 Remote Link This issue links to "Page (Confluence)" [ 13619 ] This issue links to "Page (Confluence)" [ 13619 ]

## People

• Assignee:
Nate Lust
Reporter:
Jim Bosch
Reviewers:
Russell Owen
Watchers:
Dominique Boutigny, Hsin-Fang Chiang, Jim Bosch, John Swinbank, Nate Lust, Paul Price, Russell Owen