Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-1945

HSC backport: multiband processing for coadds

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None
    • Story Points:
      8
    • Sprint:
      Science Pipelines DM-S15-3, Science Pipelines DM-S15-4
    • Team:
      Data Release Production

      Description

      This issue includes transferring changesets from many HSC issues:

      • HSC-1060
      • HSC-1064
      • HSC-1065
      • HSC-1061

      Most of this is in multiBand.py in pipe_tasks, but there are scattered changes elsewhere (including updates to camera mappers to include the new datasets, for which we'll need to modify more than just obs_subaru).

      However, before we make these changes, we'll need to open an RFC to gather comments on the design of this task. We should qualify there that this is not a long-term plan for consistent multiband processing (which we'll be starting to design on DM-1908), but a step towards better processing in the interim.

      Note: while I've assigned this to Lauren MacArthur, as I think it will be very helpful for her to get familiar with this code by doing the transfers, the RFC will have to involve a collaboration with Jim Bosch, Paul Price, and Bob Armstrong, as we can't expect someone who wasn't involved in the design to be able to write a document justifying it.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            It turns out that HSC-1061 was abandoned as the issue there was being accounted for in HSC-1063. This latter issue was also merged prior to HSC-1065, so the new order of backporting for this ticket is:

            Show
            lauren Lauren MacArthur added a comment - - edited It turns out that HSC-1061 was abandoned as the issue there was being accounted for in HSC-1063 . This latter issue was also merged prior to HSC-1065 , so the new order of backporting for this ticket is: HSC-1060 HSC-1064 HSC-1063 (omitting hscPipe changes) HSC-1065
            Hide
            lauren Lauren MacArthur added a comment - - edited

            I also included the following commits from HSC-1075

            commit fc06788de8c526612d97fb79011c1fd4292ce72a
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Dec 19 16:32:10 2014 -0500
             
                Use short names for filters in merge flag fields
             
             python/lsst/pipe/tasks/multiBand.py | 17 +++++++++++++++--
             1 file changed, 15 insertions(+), 2 deletions(-)
            

            commit 9606fd15029351680d6ce9940c678df5533537bc
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Thu Dec 18 13:50:05 2014 -0500
             
                Customize getSchemaCatalogs in merge task to save peak schema
             
             python/lsst/pipe/tasks/multiBand.py | 15 +++++++++++++--
             1 file changed, 13 insertions(+), 2 deletions(-)
            

            commit 7405def43d11b8976794c51ccbbea8eeb6170b35
            Author: Jim Bosch <jbosch@astro.princeton.edu>
            Date:   Fri Dec 19 12:40:03 2014 -0500
             
                Load peak schema and pass it to the deblender in multi-band processing
             
             python/lsst/pipe/tasks/multiBand.py | 10 +++++++---
             1 file changed, 7 insertions(+), 3 deletions(-)
            

            Show
            lauren Lauren MacArthur added a comment - - edited I also included the following commits from HSC-1075 commit fc06788de8c526612d97fb79011c1fd4292ce72a Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Dec 19 16:32:10 2014 -0500   Use short names for filters in merge flag fields   python/lsst/pipe/tasks/multiBand.py | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) commit 9606fd15029351680d6ce9940c678df5533537bc Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Thu Dec 18 13:50:05 2014 -0500   Customize getSchemaCatalogs in merge task to save peak schema   python/lsst/pipe/tasks/multiBand.py | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) commit 7405def43d11b8976794c51ccbbea8eeb6170b35 Author: Jim Bosch <jbosch@astro.princeton.edu> Date: Fri Dec 19 12:40:03 2014 -0500   Load peak schema and pass it to the deblender in multi-band processing   python/lsst/pipe/tasks/multiBand.py | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
            Hide
            lauren Lauren MacArthur added a comment -

            Also added the following commit from HSC-982 (this was delayed from DM-245 until a more appropriate time...which is now):

            commit 8f87ad50490865b5a9b2c1411b2c5a4d7f75d060
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Sep 2 19:56:21 2014 -0400
             
                config: factor out aperture settings, Kron, CModel activation
                
                Also sets forcedPhotCcd apertures to the standard set.
             
             config/apertures.py       | 10 ++++++++++
             config/cmodel.py          |  8 ++++++++
             config/forcedPhotCcd.py   | 20 ++++++--------------
             config/forcedPhotCoadd.py | 28 ++++++----------------------
             config/kron.py            |  8 ++++++++
             config/processCcd.py      | 47 ++++++++++-------------------------------------
             config/processCoadd.py    | 28 +++++-----------------------
            

            Show
            lauren Lauren MacArthur added a comment - Also added the following commit from HSC-982 (this was delayed from DM-245 until a more appropriate time...which is now): commit 8f87ad50490865b5a9b2c1411b2c5a4d7f75d060 Author: Paul Price <price@astro.princeton.edu> Date: Tue Sep 2 19:56:21 2014 -0400   config: factor out aperture settings, Kron, CModel activation Also sets forcedPhotCcd apertures to the standard set.   config/apertures.py | 10 ++++++++++ config/cmodel.py | 8 ++++++++ config/forcedPhotCcd.py | 20 ++++++-------------- config/forcedPhotCoadd.py | 28 ++++++---------------------- config/kron.py | 8 ++++++++ config/processCcd.py | 47 ++++++++++------------------------------------- config/processCoadd.py | 28 +++++-----------------------
            Hide
            lauren Lauren MacArthur added a comment -

            To keep the configs consistent, also added the following from HSC-1147:

            commit 2e8030c0661338c9eb71859749f0c750ec5032c5
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Tue Mar 17 10:25:34 2015 -0400
             
                config: set maxNumberOfPeaks for deblending on coadds
                
                In both processCoadd and the multiband processing scheme.
             
             config/measureCoaddSources.py | 2 ++
             config/processCoadd.py        | 2 ++
             2 files changed, 4 insertions(+)
            

            And, to be complete, the changesets from HSC-1192: "Set HSM as default shape" were also cherry-picked.

            Show
            lauren Lauren MacArthur added a comment - To keep the configs consistent, also added the following from HSC-1147 : commit 2e8030c0661338c9eb71859749f0c750ec5032c5 Author: Paul Price <price@astro.princeton.edu> Date: Tue Mar 17 10:25:34 2015 -0400   config: set maxNumberOfPeaks for deblending on coadds In both processCoadd and the multiband processing scheme.   config/measureCoaddSources.py | 2 ++ config/processCoadd.py | 2 ++ 2 files changed, 4 insertions(+) And, to be complete, the changesets from HSC-1192 : "Set HSM as default shape" were also cherry-picked.
            Hide
            lauren Lauren MacArthur added a comment -

            Also adding the following commit from HSC-1205 (the afw commit from that ticket was already pulled over on DM-1946):

            commit e950b97fc62002d46ee8b74ecf98c4d95767287f
            Author: Bob Armstrong <rearmstr@gmail.com>
            Date: Fri Apr 17 10:58:16 2015 -0400
            multiband: add option for flagging the same peak in different filters.
            python/lsst/pipe/tasks/multiBand.py | 8 +++++++-
            1 file changed, 7 insertions, 1 deletion
            

            Show
            lauren Lauren MacArthur added a comment - Also adding the following commit from HSC-1205 (the afw commit from that ticket was already pulled over on DM-1946 ): commit e950b97fc62002d46ee8b74ecf98c4d95767287f Author: Bob Armstrong <rearmstr@gmail.com> Date: Fri Apr 17 10:58:16 2015 -0400 multiband: add option for flagging the same peak in different filters. python/lsst/pipe/tasks/multiBand.py | 8 +++++++- 1 file changed, 7 insertions, 1 deletion
            Hide
            lauren Lauren MacArthur added a comment -

            Would you mind looking at this? I have tested it in the context of running it on a coadd I made with some HSC data. All of

            • detectCoaddSources.py
            • mergeCoaddDetections.py
            • measureCoaddSources.py
            • mergeCoaddMeasurements.py
              ran through to completion. All output products can be found on tiger at:
              /tigress/HSC/HSC/rerun/lauren/DM-1907/

            This requires DM-2606 to run as it uses the afw footprint fixes there, so the above also serves as a test of those fixes as well.

            I ran a successful Buildbot using both ticket branches and it passed.

            All the changes are in u/lauren/DM-1945 branches in 8 repos (see link on right panel).

            Show
            lauren Lauren MacArthur added a comment - Would you mind looking at this? I have tested it in the context of running it on a coadd I made with some HSC data. All of detectCoaddSources.py mergeCoaddDetections.py measureCoaddSources.py mergeCoaddMeasurements.py ran through to completion. All output products can be found on tiger at: /tigress/HSC/HSC/rerun/lauren/ DM-1907 / This requires DM-2606 to run as it uses the afw footprint fixes there, so the above also serves as a test of those fixes as well. I ran a successful Buildbot using both ticket branches and it passed. All the changes are in u/lauren/ DM-1945 branches in 8 repos (see link on right panel).
            Hide
            jbosch Jim Bosch added a comment -

            Review complete. Everything here looks great, with only minor comments on PRs for meas_base, obs_cfht, and pipe_tasks.

            Show
            jbosch Jim Bosch added a comment - Review complete. Everything here looks great, with only minor comments on PRs for meas_base, obs_cfht, and pipe_tasks.
            Hide
            swinbank John Swinbank added a comment -

            Lauren MacArthur – when you get chance, could you add a few words describing what's been done here to the release notes at https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+S15+release+notes? Thanks!

            Show
            swinbank John Swinbank added a comment - Lauren MacArthur – when you get chance, could you add a few words describing what's been done here to the release notes at https://confluence.lsstcorp.org/display/DM/Data+Release+Production+WIP+S15+release+notes? Thanks!

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel