Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5288

Port HSC pipe_tasks changesets to LSST

    XMLWordPrintable

    Details

      Description

      We identified in DM-5162 some changesets that still need to be ported from HSC to LSST:

      • 31ab5f02f7722650ad0a0eb4e2f7f8b3e0073366, 0c9a4a06bfb34ed26c72109131ef9f4a8c8f237a: multiBand: save background-subtracted coadd as deepCoadd_calexp
      • e99e140feafe28e6f034143e8ee2ae58e9a9358d: Rejig interface for DetectCoaddSourcesTask to provide non-dataRef-centric API
      • 829ee0cdd605ed027af1fada4446b715d9a5180d: multiband: activate sky objects
      • MeasureMergedCoaddSources.doMatchSources defaults to False
      • ProcessImageConfig.doWriteHeavyFootprintsInSources defaults to False ?
      • 56666e8feba6893ac95fd4982d3e0daf6baf2d34: WcsSelectImagesTask: catch imagePoly is None

      We also noticed some differences:

      • * CalibrateConfig.setDefaults doesn't call parent
      • CalibrateTask.run isn't returning apCorrMap
      • reserveFraction=-1 instead of 0.2

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Need RFC-160 approved before we can change names of coadd images (for the multiband commits), and there's some overlap with DM-5322.

            Show
            price Paul Price added a comment - Need RFC-160 approved before we can change names of coadd images (for the multiband commits), and there's some overlap with DM-5322 .
            Hide
            price Paul Price added a comment -

            David Reiss, would you mind reviewing this please? There's a bunch of small changes in pipe_tasks brought over from HSC, with small consequential changes in meas_base and a bunch of obs packages.

            It's passed Jenkins (before I rebased, so the SHAs are different from the submission).

            price@price-laptop:~/LSST/pipe/tasks (tickets/DM-5288=) $ git sub
            commit a92bf0c8d3f307ddd82f04c206952eae3b592bbe
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Oct 20 14:54:25 2015 -0400
             
                multiBand: save background-subtracted coadd as deepCoadd_calexp
                
                assembleCoadd.py writes deepCoadd, but we need the background-subtracted
                version for measurement.
             
             python/lsst/pipe/tasks/mocks/simpleMapper.py |  4 ++--
             python/lsst/pipe/tasks/multiBand.py          | 19 ++++++++++++-------
             2 files changed, 14 insertions(+), 9 deletions(-)
             
            commit 35ce14b83a5db9219a1e8a2877a091226b4f54a3
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Feb 26 20:18:07 2015 -0500
             
                DetectCoaddSourcesTask: return results
                
                This allows callers access to the results without grabbing them
                from the butler.
             
             python/lsst/pipe/tasks/multiBand.py | 1 +
             1 file changed, 1 insertion(+)
             
            commit 2b1a8b8a959728fce381c777e78216b4dac4d479
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Nov 3 14:14:02 2015 -0500
             
                multiband: activate sky objects
             
             python/lsst/pipe/tasks/multiBand.py | 2 +-
             1 file changed, 1 insertion(+), 1 deletion(-)
             
            commit cafc5573fec559c05c613ecdc060c63c0a7fb9df
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Wed Jul 24 03:16:41 2013 +0900
             
                WcsSelectImagesTask: catch imagePoly is None
                
                The polygon returned from lsst.geom.convexHull may be None if
                the inputs are not suitable for generating a polygon (e.g., they are
                all the same point).  This can happen if the Wcs is not good.
             
             python/lsst/pipe/tasks/selectImages.py | 3 +++
             1 file changed, 3 insertions(+)
             
            commit 0bc57d8fe72ec4ff48f9aa6898abeb3c5b77ecc7
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 13:56:19 2016 -0400
             
                multiband: activate astrometry
                
                It's useful to have matches to reference sources.
                We use the default AstrometryTask now because the distortion problems
                are fixed.
             
             python/lsst/pipe/tasks/multiBand.py | 6 +++---
             tests/testCoadds.py                 | 1 +
             2 files changed, 4 insertions(+), 3 deletions(-)
             
             
            price@price-laptop:~/LSST/meas/base (tickets/DM-5288=) $ git sub
            commit a644a945388c3da9882e849e7cfc1bf31eb78535
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:42:10 2016 -0400
             
                forcedPhotCoadd: adapt to rename of background-subtracted coadd
                
                It's now "deepCoadd_calexp" instead of "deepCoadd_calexp_det".
             
             python/lsst/meas/base/forcedPhotCoadd.py | 2 +-
             1 file changed, 1 insertion(+), 1 deletion(-)
             
             
            price@price-laptop:~/LSST/obs/cfht (tickets/DM-5288=) $ git sub
            commit dc1a4d43d56db316be32d267b922aa021d9b1676
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:05:52 2016 -0400
             
                MegacamMapper: adapt to multiband dataset name changes
                
                deepCoadd_calexp_det is now deepCoadd_calexp, and
                deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.
             
             policy/MegacamMapper.paf | 18 +-----------------
             1 file changed, 1 insertion(+), 17 deletions(-)
             
             
            price@price-laptop:~/LSST/obs/decam (tickets/DM-5288=) $ git sub
            commit 7deaf29275024b86fa94377d3c356ac29ed2c264
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:04:06 2016 -0400
             
                DecamMapper: adapt to multiband dataset name changes
                
                deepCoadd_calexp_det is now deepCoadd_calexp, and
                deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.
             
             policy/DecamMapper.paf | 13 ++-----------
             1 file changed, 2 insertions(+), 11 deletions(-)
             
             
            price@price-laptop:~/LSST/obs/lsstSim (tickets/DM-5288=) $ git sub
            commit 06064d799134404c0095e407ced04526e7bcfc52
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:07:45 2016 -0400
             
                LsstSimMapper: adapt to multiband dataset name changes
                
                deepCoadd_calexp_det is now deepCoadd_calexp, and
                deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.
             
             policy/LsstSimMapper.paf | 19 +------------------
             1 file changed, 1 insertion(+), 18 deletions(-)
             
             
            price@price-laptop:~/LSST/obs/sdss (tickets/DM-5288=) $ git sub
            commit d7b1456abdd0c1f4d13e852ac23c579ea4a96c2d
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:10:21 2016 -0400
             
                SdssMapper: adapt to multiband dataset name changes
                
                deepCoadd_calexp_det is now deepCoadd_calexp, and
                deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.
             
             policy/SdssMapper.paf | 19 +------------------
             1 file changed, 1 insertion(+), 18 deletions(-)
             
             
            price@price-laptop:~/LSST/obs/subaru (tickets/DM-5288 %=) $ git sub
            commit 3b094d67c96c1b131e40e973a8f79f2cb02efe32
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 29 14:01:46 2016 -0400
             
                HscMapper: adapt to multiband dataset name changes
                
                deepCoadd_calexp_det is now deepCoadd_calexp, and
                deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.
             
             policy/HscMapper.paf | 20 +-------------------
             1 file changed, 1 insertion(+), 19 deletions(-)
            

            Show
            price Paul Price added a comment - David Reiss , would you mind reviewing this please? There's a bunch of small changes in pipe_tasks brought over from HSC, with small consequential changes in meas_base and a bunch of obs packages. It's passed Jenkins (before I rebased, so the SHAs are different from the submission). price@price-laptop:~/LSST/pipe/tasks (tickets/DM-5288=) $ git sub commit a92bf0c8d3f307ddd82f04c206952eae3b592bbe Author: Paul Price <price@astro.princeton.edu> Date: Tue Oct 20 14:54:25 2015 -0400   multiBand: save background-subtracted coadd as deepCoadd_calexp assembleCoadd.py writes deepCoadd, but we need the background-subtracted version for measurement.   python/lsst/pipe/tasks/mocks/simpleMapper.py | 4 ++-- python/lsst/pipe/tasks/multiBand.py | 19 ++++++++++++------- 2 files changed, 14 insertions(+), 9 deletions(-)   commit 35ce14b83a5db9219a1e8a2877a091226b4f54a3 Author: Paul Price <price@astro.princeton.edu> Date: Thu Feb 26 20:18:07 2015 -0500   DetectCoaddSourcesTask: return results This allows callers access to the results without grabbing them from the butler.   python/lsst/pipe/tasks/multiBand.py | 1 + 1 file changed, 1 insertion(+)   commit 2b1a8b8a959728fce381c777e78216b4dac4d479 Author: Paul Price <price@astro.princeton.edu> Date: Tue Nov 3 14:14:02 2015 -0500   multiband: activate sky objects   python/lsst/pipe/tasks/multiBand.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)   commit cafc5573fec559c05c613ecdc060c63c0a7fb9df Author: Paul Price <price@astro.princeton.edu> Date: Wed Jul 24 03:16:41 2013 +0900   WcsSelectImagesTask: catch imagePoly is None The polygon returned from lsst.geom.convexHull may be None if the inputs are not suitable for generating a polygon (e.g., they are all the same point). This can happen if the Wcs is not good.   python/lsst/pipe/tasks/selectImages.py | 3 +++ 1 file changed, 3 insertions(+)   commit 0bc57d8fe72ec4ff48f9aa6898abeb3c5b77ecc7 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 13:56:19 2016 -0400   multiband: activate astrometry It's useful to have matches to reference sources. We use the default AstrometryTask now because the distortion problems are fixed.   python/lsst/pipe/tasks/multiBand.py | 6 +++--- tests/testCoadds.py | 1 + 2 files changed, 4 insertions(+), 3 deletions(-)     price@price-laptop:~/LSST/meas/base (tickets/DM-5288=) $ git sub commit a644a945388c3da9882e849e7cfc1bf31eb78535 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:42:10 2016 -0400   forcedPhotCoadd: adapt to rename of background-subtracted coadd It's now "deepCoadd_calexp" instead of "deepCoadd_calexp_det".   python/lsst/meas/base/forcedPhotCoadd.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)     price@price-laptop:~/LSST/obs/cfht (tickets/DM-5288=) $ git sub commit dc1a4d43d56db316be32d267b922aa021d9b1676 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:05:52 2016 -0400   MegacamMapper: adapt to multiband dataset name changes deepCoadd_calexp_det is now deepCoadd_calexp, and deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.   policy/MegacamMapper.paf | 18 +----------------- 1 file changed, 1 insertion(+), 17 deletions(-)     price@price-laptop:~/LSST/obs/decam (tickets/DM-5288=) $ git sub commit 7deaf29275024b86fa94377d3c356ac29ed2c264 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:04:06 2016 -0400   DecamMapper: adapt to multiband dataset name changes deepCoadd_calexp_det is now deepCoadd_calexp, and deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.   policy/DecamMapper.paf | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-)     price@price-laptop:~/LSST/obs/lsstSim (tickets/DM-5288=) $ git sub commit 06064d799134404c0095e407ced04526e7bcfc52 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:07:45 2016 -0400   LsstSimMapper: adapt to multiband dataset name changes deepCoadd_calexp_det is now deepCoadd_calexp, and deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.   policy/LsstSimMapper.paf | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-)     price@price-laptop:~/LSST/obs/sdss (tickets/DM-5288=) $ git sub commit d7b1456abdd0c1f4d13e852ac23c579ea4a96c2d Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:10:21 2016 -0400   SdssMapper: adapt to multiband dataset name changes deepCoadd_calexp_det is now deepCoadd_calexp, and deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.   policy/SdssMapper.paf | 19 +------------------ 1 file changed, 1 insertion(+), 18 deletions(-)     price@price-laptop:~/LSST/obs/subaru (tickets/DM-5288 %=) $ git sub commit 3b094d67c96c1b131e40e973a8f79f2cb02efe32 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 29 14:01:46 2016 -0400   HscMapper: adapt to multiband dataset name changes deepCoadd_calexp_det is now deepCoadd_calexp, and deepCoadd_calexp_detBackground is now deepCoadd_calexp_background.   policy/HscMapper.paf | 20 +------------------- 1 file changed, 1 insertion(+), 19 deletions(-)
            Hide
            price Paul Price added a comment -

            Oh, I should mention I didn't change anything in CalibrateTask, because of the recent big refactoring to that code on the LSST side which I think addressed my two little concerns. And I haven't changed the reserveFraction because that has implications so it needs to be agreed upon.

            Show
            price Paul Price added a comment - Oh, I should mention I didn't change anything in CalibrateTask , because of the recent big refactoring to that code on the LSST side which I think addressed my two little concerns. And I haven't changed the reserveFraction because that has implications so it needs to be agreed upon.
            Hide
            swinbank John Swinbank added a comment -

            Paul Price – what's the process for agreeing on changes to reserveFraction? Is this something we are happy to override in obs_subaru if HSC wants a different value, or are you suggesting we need an RFC?

            Show
            swinbank John Swinbank added a comment - Paul Price – what's the process for agreeing on changes to reserveFraction ? Is this something we are happy to override in obs_subaru if HSC wants a different value, or are you suggesting we need an RFC?
            Hide
            price Paul Price added a comment -

            I don't know if it needs an RFC, but we definitely need wider agreement than just me. Maybe we should just RFC it.

            Show
            price Paul Price added a comment - I don't know if it needs an RFC, but we definitely need wider agreement than just me. Maybe we should just RFC it.
            Hide
            reiss David Reiss added a comment -

            Paul Price I have finished the review and have no more questions, so feel free to merge.

            Show
            reiss David Reiss added a comment - Paul Price I have finished the review and have no more questions, so feel free to merge.
            Hide
            price Paul Price added a comment -

            Thanks David Reiss!

            We discussed the reserveFraction at Princeton, and agreed to add this as a Subaru override rather than changing the default setting. I've added one more commit in obs_subaru, so please let me know if there are any problems with that:

            price@price-laptop:~/LSST/obs/subaru (tickets/DM-5288=) $ git --no-pager log -1 --patch
            commit 708a24936dd2e164d0c1373a5357ad32747ee71a
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Thu Mar 31 15:33:11 2016 -0400
             
                config: reserve PSF candidates for QA
                
                For HSC SSP, we have been reserving 20% of PSF candidates for QA.
                This setting fell out when the feature was ported to LSST.
             
            diff --git a/config/processCcd.py b/config/processCcd.py
            index 72e99a5..692bc7c 100644
            --- a/config/processCcd.py
            +++ b/config/processCcd.py
            @@ -15,6 +15,7 @@ config.charImage.detectAndMeasure.detection.background.binSize = 128
             config.calibrate.detectAndMeasure.detection.background.binSize = 128
             
             # PSF determination
            +config.charImage.measurePsf.reserveFraction = 0.2
             config.charImage.measurePsf.starSelector['objectSize'].sourceFluxField = 'base_PsfFlux_flux'
             try:
                 import lsst.meas.extensions.psfex.psfexPsfDeterminer
            

            I'll run this through Jenkins before merging.

            Show
            price Paul Price added a comment - Thanks David Reiss ! We discussed the reserveFraction at Princeton, and agreed to add this as a Subaru override rather than changing the default setting. I've added one more commit in obs_subaru, so please let me know if there are any problems with that: price@price-laptop:~/LSST/obs/subaru (tickets/DM-5288=) $ git --no-pager log -1 --patch commit 708a24936dd2e164d0c1373a5357ad32747ee71a Author: Paul Price <price@astro.princeton.edu> Date: Thu Mar 31 15:33:11 2016 -0400   config: reserve PSF candidates for QA For HSC SSP, we have been reserving 20% of PSF candidates for QA. This setting fell out when the feature was ported to LSST.   diff --git a/config/processCcd.py b/config/processCcd.py index 72e99a5..692bc7c 100644 --- a/config/processCcd.py +++ b/config/processCcd.py @@ -15,6 +15,7 @@ config.charImage.detectAndMeasure.detection.background.binSize = 128 config.calibrate.detectAndMeasure.detection.background.binSize = 128 # PSF determination +config.charImage.measurePsf.reserveFraction = 0.2 config.charImage.measurePsf.starSelector['objectSize'].sourceFluxField = 'base_PsfFlux_flux' try: import lsst.meas.extensions.psfex.psfexPsfDeterminer I'll run this through Jenkins before merging.
            Hide
            reiss David Reiss added a comment -

            Seems fine to me.

            Show
            reiss David Reiss added a comment - Seems fine to me.
            Hide
            price Paul Price added a comment -

            Thanks again, David Reiss.

            After a final Jenkins run, I've merged everything to master.

            Show
            price Paul Price added a comment - Thanks again, David Reiss . After a final Jenkins run , I've merged everything to master.

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              David Reiss
              Watchers:
              David Reiss, John Swinbank, Lauren MacArthur, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.