Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13388

Enable visit-level sky subtraction for HSC by default

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ci_hsc, obs_subaru
    • Labels:
      None
    • Story Points:
      0.5
    • Team:
      External

      Description

      Turn on visit-level sky subtraction by default for HSC, which I believe means updating the coaddition configuration to expect its outputs to be available.

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Also need to update ci_hsc to perform the correction.

            Show
            price Paul Price added a comment - Also need to update ci_hsc to perform the correction.
            Hide
            price Paul Price added a comment - - edited

            New calibs (CALIB-20180117) are being transferred from Princeton to NCSA via Globus.

            Show
            price Paul Price added a comment - - edited New calibs ( CALIB-20180117 ) are being transferred from Princeton to NCSA via Globus .
            Hide
            price Paul Price added a comment -

            price@pap-laptop:~/LSST/obs_subaru (tickets/DM-13388=) $ git sub-patch
            commit 11938cd10b86486890dbdc8cd536d122ac03e6a1 (HEAD -> tickets/DM-13388, origin/tickets/DM-13388)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 14:55:33 2018 -0500
             
                config: activate sky correction by default
                
                To ensure everyone is using the latest procedures and code, enable
                sky correction by default for HSC.
             
            diff --git a/config/hsc/makeCoaddTempExp.py b/config/hsc/makeCoaddTempExp.py
            index 69afb58..d2aa276 100644
            --- a/config/hsc/makeCoaddTempExp.py
            +++ b/config/hsc/makeCoaddTempExp.py
            @@ -1 +1,2 @@
             config.doApplyUberCal = True
            +config.doApplySkyCorr = True
             
             
            ^Cprice@pap-laptop:~/LSST/ci_hsc (tickets/DM-13388=) $ git sub
            commit 9b971f7d332c706d28f7359bd8a63c0b00762408
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 15:35:15 2018 -0500
             
                validate: make more standard-compliant
                
                Include __future__ imports and two lines between class definitions.
             
             python/lsst/ci/hsc/validate.py | 12 ++++++++++++
             1 file changed, 12 insertions(+)
             
            commit 69c96a3d1f88a704b3e3d36fdb8d3fefe44b45ba
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 15:45:56 2018 -0500
             
                allow validate function to be used like butler
                
                The butler allows either butler.get("foo", dataId) or
                butler.get("foo", key=value); let validate do the same
                for optimal user convenience.
             
             SConstruct | 4 ++--
             1 file changed, 2 insertions(+), 2 deletions(-)
             
            commit c07576a8efed4d38d58dcbcf47fe9ca574646f2c
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 15:52:23 2018 -0500
             
                add sky correction
                
                The sky correction runs a pipe_drivers script that corrects the sky
                subtraction. We want this to run by default, and hence we should test
                it here. We explicitly use --cores=1 because the parallelism is provided
                by scons. This should be OK because we don't have full visits to load
                into the memory of a single process, but only a few CCDs.
             
             SConstruct                     | 24 +++++++++++++++++++++++-
             python/lsst/ci/hsc/validate.py | 12 ++++++++----
             2 files changed, 31 insertions(+), 5 deletions(-)
             
            commit 52b1bfdf05f3d5ea903525faabb644f20a945492
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 16:17:47 2018 -0500
             
                add sky correction files
                
                We need them now.
                
                validStart, validEnd dates set to 2013-06-01, 2013-12-01 but sky frames
                are actually from 2014 (because we don't have any SSP_WIDE data taken
                during commissioning). That shouldn't make much difference, since the
                sky correction shouldn't change much.
             
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-000.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-001.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-004.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-005.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-006.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-010.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-011.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-012.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-016.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-017.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-018.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-022.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-023.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-024.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-025.fits  |   3 +++
             .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-100.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-000.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-001.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-004.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-005.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-006.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-010.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-011.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-012.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-016.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-017.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-018.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-022.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-023.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-024.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-025.fits  |   3 +++
             .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-100.fits  |   3 +++
             CALIB/calibRegistry.sqlite3                         | Bin 22528 -> 29696 bytes
             33 files changed, 96 insertions(+)
             
            commit 084b5127086659f6664174a9d3ed7c10229da83a (HEAD -> tickets/DM-13388, origin/tickets/DM-13388)
            Author: Paul Price <price@astro.princeton.edu>
            Date:   Mon Jan 29 19:29:16 2018 -0500
             
                fix installation of transmission curves
                
                Without getExecutable, this was vulnerable to SIP on OSX.
             
             SConstruct | 2 +-
             1 file changed, 1 insertion(+), 1 deletion(-)
            

            Show
            price Paul Price added a comment - price@pap-laptop:~/LSST/obs_subaru (tickets/DM-13388=) $ git sub-patch commit 11938cd10b86486890dbdc8cd536d122ac03e6a1 (HEAD -> tickets/DM-13388, origin/tickets/DM-13388) Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 14:55:33 2018 -0500   config: activate sky correction by default To ensure everyone is using the latest procedures and code, enable sky correction by default for HSC.   diff --git a/config/hsc/makeCoaddTempExp.py b/config/hsc/makeCoaddTempExp.py index 69afb58..d2aa276 100644 --- a/config/hsc/makeCoaddTempExp.py +++ b/config/hsc/makeCoaddTempExp.py @@ -1 +1,2 @@ config.doApplyUberCal = True +config.doApplySkyCorr = True     ^Cprice@pap-laptop:~/LSST/ci_hsc (tickets/DM-13388=) $ git sub commit 9b971f7d332c706d28f7359bd8a63c0b00762408 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:35:15 2018 -0500   validate: make more standard-compliant Include __future__ imports and two lines between class definitions.   python/lsst/ci/hsc/validate.py | 12 ++++++++++++ 1 file changed, 12 insertions(+)   commit 69c96a3d1f88a704b3e3d36fdb8d3fefe44b45ba Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:45:56 2018 -0500   allow validate function to be used like butler The butler allows either butler.get("foo", dataId) or butler.get("foo", key=value); let validate do the same for optimal user convenience.   SConstruct | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)   commit c07576a8efed4d38d58dcbcf47fe9ca574646f2c Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:52:23 2018 -0500   add sky correction The sky correction runs a pipe_drivers script that corrects the sky subtraction. We want this to run by default, and hence we should test it here. We explicitly use --cores=1 because the parallelism is provided by scons. This should be OK because we don't have full visits to load into the memory of a single process, but only a few CCDs.   SConstruct | 24 +++++++++++++++++++++++- python/lsst/ci/hsc/validate.py | 12 ++++++++---- 2 files changed, 31 insertions(+), 5 deletions(-)   commit 52b1bfdf05f3d5ea903525faabb644f20a945492 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 16:17:47 2018 -0500   add sky correction files We need them now. validStart, validEnd dates set to 2013-06-01, 2013-12-01 but sky frames are actually from 2014 (because we don't have any SSP_WIDE data taken during commissioning). That shouldn't make much difference, since the sky correction shouldn't change much.   .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-000.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-001.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-004.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-005.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-006.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-010.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-011.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-012.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-016.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-017.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-018.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-022.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-023.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-024.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-025.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-100.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-000.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-001.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-004.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-005.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-006.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-010.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-011.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-012.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-016.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-017.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-018.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-022.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-023.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-024.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-025.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-100.fits | 3 +++ CALIB/calibRegistry.sqlite3 | Bin 22528 -> 29696 bytes 33 files changed, 96 insertions(+)   commit 084b5127086659f6664174a9d3ed7c10229da83a (HEAD -> tickets/DM-13388, origin/tickets/DM-13388) Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 19:29:16 2018 -0500   fix installation of transmission curves Without getExecutable, this was vulnerable to SIP on OSX.   SConstruct | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
            Hide
            price Paul Price added a comment -

            Thanks for volunteering to review, Jim Bosch.

            Show
            price Paul Price added a comment - Thanks for volunteering to review, Jim Bosch .
            Hide
            jbosch Jim Bosch added a comment -

            Looks good!  Minor comments on the PR.

            Thanks for fixing my ci_hsc failures!

             

            Show
            jbosch Jim Bosch added a comment - Looks good!  Minor comments on the PR. Thanks for fixing my ci_hsc failures!  
            Hide
            price Paul Price added a comment -

            Jenkins is green apart from centos6-py2 which failed with a system error.

            Show
            price Paul Price added a comment - Jenkins is green apart from centos6-py2 which failed with a system error.
            Hide
            price Paul Price added a comment -

            Merged to master.

            Show
            price Paul Price added a comment - Merged to master.
            Hide
            price Paul Price added a comment -
            Show
            price Paul Price added a comment - Notification posted to CLO: https://community.lsst.org/t/hsc-defaults-now-require-sky-correction/2631

              People

              • Assignee:
                price Paul Price
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Jim Bosch
                Watchers:
                Jim Bosch, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel