Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3942

Preliminaries for the LSST vs. HSC pipeline comparison through single-frame processing

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Science Pipelines
    • Labels:
      None
    • Story Points:
      14
    • Sprint:
      Science Pipelines DM-W16-1, Science Pipelines DM-W16-2
    • Team:
      Data Release Production

      Description

      This ticket is in preparation for DM-2984, which is to run both the HSC and LSST pipelines on 2-3 visits of HSC data, and do a detailed comparison of the science quality and robustness for the single-frame processing (ProcessCcdTask) stage only. Essentially, it is a detailed audit of all the functionality currently used in HSC single-frame processing and ensuring all relevant features have been pulled over to the LSST stack such that the comparisons to be done in DM-2984 will be meaningful and informative.

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment - - edited

            I have identified the following tickets and standalone commits that need pulling over from HSC:

            obs_subaru:

            The obs_subaru commits from HSC-987:
            HscMapper: remove old filter names (W-S-*) as independent filters
            HscMapper: add filters NB0515, NB0656.
            HscMapper: set default filter name
            HscMapper: make UNRECOGNISED the canonical name for unrecognised filters

            Standalone:
            HscMapper: add different spellings of unrecognised filter names

            HSC-1161: Add mapper support for filter NB816
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/1e085f9b655772590f7c0b74486ba8ada698537d

            HSC-1160: Mark amplifier on CCD 0 as bad
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/820d11753afb3458c8cdcef28adc3897faca8c03

            HSC-1147: Large footprint due to satellite trail
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/2e8030c0661338c9eb71859749f0c750ec5032c5 NO-OP

            HSC-1201: reduceBias.py fails in isr.roughZeroPoint
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/c9081ebfedffcf33bd23990f6e1ef786da89b54a

            HSC-1208: Add nominal magzero for NB filters
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/de96531db3f4a3990636338c016ab7d10d3704f9

            HSC-1231:ValueError from over-exposed frame in flat construction
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/dadfd7e62b1e9849212bdac3fbc4329407fe298d
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/d6fe6cf5c4ecadebd5a344d163e1f1e60137c7e4 BLOCKED by HSC-1095: will address on DM-2984

            Standalone:
            config: remove deblender overrides

            Standalone:
            config: reduce deblend.maxFootprintArea only for processCcd

            HSC-1289: Footprints still being read when SOURCE_IO_NO_FOOTPRINTS flag is passed
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/78004d16fb9fa84bcdf893e84992612af4961178

            HSC-1310: Activate fringe subtraction for NB921
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/52feb4f8422bec1f49d4731ee983ab7a535f525e

            HSC-1296: Add support for new NB filters
            https://github.com/HyperSuprime-Cam/obs_subaru/commit/06a16ace77f6bc85abc895d528b249e77c32db48

            Show
            lauren Lauren MacArthur added a comment - - edited I have identified the following tickets and standalone commits that need pulling over from HSC: obs_subaru : The obs_subaru commits from HSC-987 : HscMapper: remove old filter names (W-S-*) as independent filters HscMapper: add filters NB0515, NB0656. HscMapper: set default filter name HscMapper: make UNRECOGNISED the canonical name for unrecognised filters Standalone: HscMapper: add different spellings of unrecognised filter names HSC-1161 : Add mapper support for filter NB816 https://github.com/HyperSuprime-Cam/obs_subaru/commit/1e085f9b655772590f7c0b74486ba8ada698537d HSC-1160 : Mark amplifier on CCD 0 as bad https://github.com/HyperSuprime-Cam/obs_subaru/commit/820d11753afb3458c8cdcef28adc3897faca8c03 HSC-1147 : Large footprint due to satellite trail https://github.com/HyperSuprime-Cam/obs_subaru/commit/2e8030c0661338c9eb71859749f0c750ec5032c5 NO-OP HSC-1201 : reduceBias.py fails in isr.roughZeroPoint https://github.com/HyperSuprime-Cam/obs_subaru/commit/c9081ebfedffcf33bd23990f6e1ef786da89b54a HSC-1208 : Add nominal magzero for NB filters https://github.com/HyperSuprime-Cam/obs_subaru/commit/de96531db3f4a3990636338c016ab7d10d3704f9 HSC-1231 :ValueError from over-exposed frame in flat construction https://github.com/HyperSuprime-Cam/obs_subaru/commit/dadfd7e62b1e9849212bdac3fbc4329407fe298d https://github.com/HyperSuprime-Cam/obs_subaru/commit/d6fe6cf5c4ecadebd5a344d163e1f1e60137c7e4 BLOCKED by HSC-1095: will address on DM-2984 Standalone: config: remove deblender overrides Standalone: config: reduce deblend.maxFootprintArea only for processCcd HSC-1289 : Footprints still being read when SOURCE_IO_NO_FOOTPRINTS flag is passed https://github.com/HyperSuprime-Cam/obs_subaru/commit/78004d16fb9fa84bcdf893e84992612af4961178 HSC-1310 : Activate fringe subtraction for NB921 https://github.com/HyperSuprime-Cam/obs_subaru/commit/52feb4f8422bec1f49d4731ee983ab7a535f525e HSC-1296 : Add support for new NB filters https://github.com/HyperSuprime-Cam/obs_subaru/commit/06a16ace77f6bc85abc895d528b249e77c32db48
            Hide
            lauren Lauren MacArthur added a comment -

            John, would you have time to give this a review? I have run processCcd.py and hscProcessCcd.py with the latest LSST (w_2015_43) and HSC (hscPipe 3.8.6c_hsc), respectively, stacks installed on tiger and have compared in detail their configs, in particular looking for any setting differences and missing functionality. I believe with these changes we are now in a very good position to move onto DM-2984 (where we will start comparing outputs, starting with the most basic measurement algorithms).

            I reran processCcd.py after making the changes on obs_subaru here and all looks fine.

            Show
            lauren Lauren MacArthur added a comment - John, would you have time to give this a review? I have run processCcd.py and hscProcessCcd.py with the latest LSST (w_2015_43) and HSC (hscPipe 3.8.6c_hsc), respectively, stacks installed on tiger and have compared in detail their configs, in particular looking for any setting differences and missing functionality. I believe with these changes we are now in a very good position to move onto DM-2984 (where we will start comparing outputs, starting with the most basic measurement algorithms). I reran processCcd.py after making the changes on obs_subaru here and all looks fine.
            Hide
            swinbank John Swinbank added a comment -

            Changes in obs_subaru look fine, with one potential caveat: in 5c4a20 I find the commit message quite confusing. I don't think this is fixing a bug in the Butler, I think it's fixing a bug (or, at least, resolving a potential issue) whereby HscMapper was losing flags. Could you reword the message to be a bit clearer that we don't expect a Butler bug to be raised/fixed in response to this, please?

            Other than that, good to merge.

            Show
            swinbank John Swinbank added a comment - Changes in obs_subaru look fine, with one potential caveat: in 5c4a20 I find the commit message quite confusing. I don't think this is fixing a bug in the Butler, I think it's fixing a bug (or, at least, resolving a potential issue) whereby HscMapper was losing flags. Could you reword the message to be a bit clearer that we don't expect a Butler bug to be raised/fixed in response to this, please? Other than that, good to merge.
            Hide
            lauren Lauren MacArthur added a comment -

            Thanks John Swinbank. I changed the commit message and added some description to it.

            Merged to master.

            Show
            lauren Lauren MacArthur added a comment - Thanks John Swinbank . I changed the commit message and added some description to it. Merged to master.
            Hide
            swinbank John Swinbank added a comment -

            I assume that quite a few of these configuration changes cause noticeable changes to the outputs. Could you add a brief not on that to the release notes, please? I don't think it's necessary to describe every individual change in detail, but some indication that end users should expect results to change if they're relying on the default configuration would be good.

            Show
            swinbank John Swinbank added a comment - I assume that quite a few of these configuration changes cause noticeable changes to the outputs. Could you add a brief not on that to the release notes , please? I don't think it's necessary to describe every individual change in detail, but some indication that end users should expect results to change if they're relying on the default configuration would be good.

              People

              • Assignee:
                lauren Lauren MacArthur
                Reporter:
                lauren Lauren MacArthur
                Reviewers:
                John Swinbank
                Watchers:
                John Swinbank, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: