[DM-13388] Enable visit-level sky subtraction for HSC by default - Jira

XML

Word

Printable

Details

Type: Story
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ci_hsc, obs_subaru
Labels:
None

Story Points:
0.5
Team:
External

Description

Turn on visit-level sky subtraction by default for HSC, which I believe means updating the coaddition configuration to expect its outputs to be available.

Attachments

Issue Links

is blocked by

DM-13387 Update calibrations in /dataset/hsc/repo for sky subtraction and transmission curves

Done

is triggered by

RFC-440 Update calibrations in /datasets/hsc/repo and enable new HSC pipeline steps

Implemented

is triggering

DM-13553 Deal with large blends

Done

relates to

DM-26950 Reprocess and Compare RC2 with Increased Sky Correction bgModel2 Values

Done

Activity

Ascending order - Click to sort in descending order

No builds found.

Jim Bosch created issue - 27/Jan/18 10:36 AM

Jim Bosch made changes - 27/Jan/18 10:36 AM

Field	Original Value	New Value
Link		This issue is triggered by ~~RFC-440~~ [ ~~RFC-440~~ ]

Jim Bosch made changes - 27/Jan/18 10:38 AM

Link

This issue is blocked by ~~DM-13387~~ [ ~~DM-13387~~ ]

Hide

Permalink

Paul Price added a comment - 27/Jan/18 12:12 PM

Also need to update ci_hsc to perform the correction.

Show

Paul Price added a comment - 27/Jan/18 12:12 PM Also need to update ci_hsc to perform the correction.

Paul Price made changes - 27/Jan/18 12:12 PM

Component/s

ci_hsc [ 13140 ]

Hide

Permalink

Paul Price added a comment - 29/Jan/18 12:34 PM - edited

New calibs (CALIB-20180117) are being transferred from Princeton to NCSA via Globus.

Show

Paul Price added a comment - 29/Jan/18 12:34 PM - edited New calibs ( CALIB-20180117 ) are being transferred from Princeton to NCSA via Globus .

Hide

Permalink

Paul Price added a comment - 29/Jan/18 7:02 PM

price@pap-laptop:~/LSST/obs_subaru (tickets/DM-13388=) $ git sub-patch

commit 11938cd10b86486890dbdc8cd536d122ac03e6a1 (HEAD -> tickets/DM-13388, origin/tickets/DM-13388)

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 14:55:33 2018 -0500

    config: activate sky correction by default

    To ensure everyone is using the latest procedures and code, enable

    sky correction by default for HSC.

diff --git a/config/hsc/makeCoaddTempExp.py b/config/hsc/makeCoaddTempExp.py

index 69afb58..d2aa276 100644

--- a/config/hsc/makeCoaddTempExp.py

+++ b/config/hsc/makeCoaddTempExp.py

@@ -1 +1,2 @@

 config.doApplyUberCal = True

+config.doApplySkyCorr = True

^Cprice@pap-laptop:~/LSST/ci_hsc (tickets/DM-13388=) $ git sub

commit 9b971f7d332c706d28f7359bd8a63c0b00762408

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 15:35:15 2018 -0500

    validate: make more standard-compliant

    Include __future__ imports and two lines between class definitions.

 python/lsst/ci/hsc/validate.py | 12 ++++++++++++

 1 file changed, 12 insertions(+)

commit 69c96a3d1f88a704b3e3d36fdb8d3fefe44b45ba

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 15:45:56 2018 -0500

    allow validate function to be used like butler

    The butler allows either butler.get("foo", dataId) or

    butler.get("foo", key=value); let validate do the same

    for optimal user convenience.

 SConstruct | 4 ++--

 1 file changed, 2 insertions(+), 2 deletions(-)

commit c07576a8efed4d38d58dcbcf47fe9ca574646f2c

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 15:52:23 2018 -0500

    add sky correction

    The sky correction runs a pipe_drivers script that corrects the sky

    subtraction. We want this to run by default, and hence we should test

    it here. We explicitly use --cores=1 because the parallelism is provided

    by scons. This should be OK because we don't have full visits to load

    into the memory of a single process, but only a few CCDs.

 SConstruct                     | 24 +++++++++++++++++++++++-

 python/lsst/ci/hsc/validate.py | 12 ++++++++----

 2 files changed, 31 insertions(+), 5 deletions(-)

commit 52b1bfdf05f3d5ea903525faabb644f20a945492

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 16:17:47 2018 -0500

    add sky correction files

    We need them now.

    validStart, validEnd dates set to 2013-06-01, 2013-12-01 but sky frames

    are actually from 2014 (because we don't have any SSP_WIDE data taken

    during commissioning). That shouldn't make much difference, since the

    sky correction shouldn't change much.

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-000.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-001.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-004.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-005.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-006.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-010.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-011.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-012.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-016.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-017.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-018.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-022.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-023.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-024.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-025.fits  |   3 +++

 .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-100.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-000.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-001.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-004.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-005.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-006.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-010.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-011.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-012.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-016.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-017.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-018.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-022.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-023.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-024.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-025.fits  |   3 +++

 .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-100.fits  |   3 +++

 CALIB/calibRegistry.sqlite3                         | Bin 22528 -> 29696 bytes

 33 files changed, 96 insertions(+)

commit 084b5127086659f6664174a9d3ed7c10229da83a (HEAD -> tickets/DM-13388, origin/tickets/DM-13388)

Author: Paul Price <price@astro.princeton.edu>

Date:   Mon Jan 29 19:29:16 2018 -0500

    fix installation of transmission curves

    Without getExecutable, this was vulnerable to SIP on OSX.

 SConstruct | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)

Show

Paul Price added a comment - 29/Jan/18 7:02 PM price@pap-laptop:~/LSST/obs_subaru (tickets/DM-13388=) $ git sub-patch commit 11938cd10b86486890dbdc8cd536d122ac03e6a1 (HEAD -> tickets/DM-13388, origin/tickets/DM-13388) Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 14:55:33 2018 -0500 config: activate sky correction by default To ensure everyone is using the latest procedures and code, enable sky correction by default for HSC. diff --git a/config/hsc/makeCoaddTempExp.py b/config/hsc/makeCoaddTempExp.py index 69afb58..d2aa276 100644 --- a/config/hsc/makeCoaddTempExp.py +++ b/config/hsc/makeCoaddTempExp.py @@ -1 +1,2 @@ config.doApplyUberCal = True +config.doApplySkyCorr = True ^Cprice@pap-laptop:~/LSST/ci_hsc (tickets/DM-13388=) $ git sub commit 9b971f7d332c706d28f7359bd8a63c0b00762408 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:35:15 2018 -0500 validate: make more standard-compliant Include __future__ imports and two lines between class definitions. python/lsst/ci/hsc/validate.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) commit 69c96a3d1f88a704b3e3d36fdb8d3fefe44b45ba Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:45:56 2018 -0500 allow validate function to be used like butler The butler allows either butler.get("foo", dataId) or butler.get("foo", key=value); let validate do the same for optimal user convenience. SConstruct | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) commit c07576a8efed4d38d58dcbcf47fe9ca574646f2c Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 15:52:23 2018 -0500 add sky correction The sky correction runs a pipe_drivers script that corrects the sky subtraction. We want this to run by default, and hence we should test it here. We explicitly use --cores=1 because the parallelism is provided by scons. This should be OK because we don't have full visits to load into the memory of a single process, but only a few CCDs. SConstruct | 24 +++++++++++++++++++++++- python/lsst/ci/hsc/validate.py | 12 ++++++++---- 2 files changed, 31 insertions(+), 5 deletions(-) commit 52b1bfdf05f3d5ea903525faabb644f20a945492 Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 16:17:47 2018 -0500 add sky correction files We need them now. validStart, validEnd dates set to 2013-06-01, 2013-12-01 but sky frames are actually from 2014 (because we don't have any SSP_WIDE data taken during commissioning). That shouldn't make much difference, since the sky correction shouldn't change much. .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-000.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-001.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-004.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-005.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-006.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-010.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-011.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-012.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-016.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-017.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-018.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-022.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-023.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-024.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-025.fits | 3 +++ .../2014-07-14/HSC-I/SKY-2014-07-14-HSC-I-100.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-000.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-001.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-004.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-005.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-006.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-010.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-011.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-012.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-016.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-017.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-018.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-022.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-023.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-024.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-025.fits | 3 +++ .../2014-11-12/HSC-R/SKY-2014-11-12-HSC-R-100.fits | 3 +++ CALIB/calibRegistry.sqlite3 | Bin 22528 -> 29696 bytes 33 files changed, 96 insertions(+) commit 084b5127086659f6664174a9d3ed7c10229da83a (HEAD -> tickets/DM-13388, origin/tickets/DM-13388) Author: Paul Price <price@astro.princeton.edu> Date: Mon Jan 29 19:29:16 2018 -0500 fix installation of transmission curves Without getExecutable, this was vulnerable to SIP on OSX. SConstruct | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

Hide

Permalink

Paul Price added a comment - 29/Jan/18 7:06 PM

Thanks for volunteering to review, Jim Bosch.

Show

Paul Price added a comment - 29/Jan/18 7:06 PM Thanks for volunteering to review, Jim Bosch .

Paul Price made changes - 29/Jan/18 7:06 PM

Reviewers		Jim Bosch [ jbosch ]
Status	To Do [ 10001 ]	In Review [ 10004 ]

Hide

Permalink

Jim Bosch added a comment - 29/Jan/18 7:30 PM

Looks good! Minor comments on the PR.

Thanks for fixing my ci_hsc failures!

Show

Jim Bosch added a comment - 29/Jan/18 7:30 PM Looks good! Minor comments on the PR. Thanks for fixing my ci_hsc failures!

Jim Bosch made changes - 29/Jan/18 7:30 PM

Status

In Review [ 10004 ]

Reviewed [ 10101 ]

Hide

Permalink

Paul Price added a comment - 30/Jan/18 8:02 AM

Jenkins is green apart from centos6-py2 which failed with a system error.

Show

Paul Price added a comment - 30/Jan/18 8:02 AM Jenkins is green apart from centos6-py2 which failed with a system error.

Hide

Permalink

Paul Price added a comment - 30/Jan/18 9:02 AM

Merged to master.

Show

Paul Price added a comment - 30/Jan/18 9:02 AM Merged to master.

Paul Price made changes - 30/Jan/18 9:02 AM

Resolution		Done [ 10000 ]
Status	Reviewed [ 10101 ]	Done [ 10002 ]

Hide

Permalink

Paul Price added a comment - 30/Jan/18 9:03 AM

Notification posted to CLO: https://community.lsst.org/t/hsc-defaults-now-require-sky-correction/2631

Show

Paul Price added a comment - 30/Jan/18 9:03 AM Notification posted to CLO: https://community.lsst.org/t/hsc-defaults-now-require-sky-correction/2631

Paul Price made changes - 14/Feb/18 11:09 AM

Link

This issue is triggering ~~DM-13553~~ [ ~~DM-13553~~ ]

John Swinbank made changes - 14/Feb/18 6:06 PM

Team

Data Release Production [ 10301 ]

External [ 12117 ]

Lee Kelvin made changes - 28/Sep/20 12:35 AM

Link

This issue relates to ~~DM-26950~~ [ ~~DM-26950~~ ]

People

Assignee:: Paul Price

Reporter:: Jim Bosch

Reviewers:: Jim Bosch

Watchers:: Jim Bosch, Paul Price

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 27/Jan/18 10:36 AM

Updated:: 28/Sep/20 12:35 AM

Resolved:: 30/Jan/18 9:02 AM

Jenkins

No builds found.

Data Management

Details

Description

Attachments

Issue Links

Activity

People

Dates

Jenkins