Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9800

Define the steps and configs for reprocessing the HSC PDR1 data on lsstvc

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Decide and document the stack version, task steps, and configs to be used in the upcoming reprocessing of the HSC PDR1 data on the Verification Cluster.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For future reference, I started coadd processing without DM-10451 (i.e. a consistent package setup). I'll make notes on the confluence page.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For future reference, I started coadd processing without DM-10451 (i.e. a consistent package setup). I'll make notes on the confluence page.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks for signing this off! I'll go ahead without DM-10451. I'll use the w_2017_17 stack + master meas_mosaic/obs_subaru/ctrl_pool as of today built with w_2017_17; effectively this is w_2017_17 + DM-10315 + DM-10449 + DM-10430.

            If I understand correctly, the CModel config change of DM-10451 matters more in coadd and beyond. We have some time before any coadd processing; I'll check the status of DM-10451 before starting coadd.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks for signing this off! I'll go ahead without DM-10451 . I'll use the w_2017_17 stack + master meas_mosaic/obs_subaru/ctrl_pool as of today built with w_2017_17; effectively this is w_2017_17 + DM-10315 + DM-10449 + DM-10430 . If I understand correctly, the CModel config change of DM-10451 matters more in coadd and beyond. We have some time before any coadd processing; I'll check the status of DM-10451 before starting coadd.
            Hide
            swinbank John Swinbank added a comment -

            Looks good to me. Per discussion with Jim Bosch, there's no need for you to block on DM-10451 (if you want to manually adopt the config changes from that ticket that's great, but it's not required). Feel free to go ahead whenever you're ready.

            Show
            swinbank John Swinbank added a comment - Looks good to me. Per discussion with Jim Bosch , there's no need for you to block on DM-10451 (if you want to manually adopt the config changes from that ticket that's great, but it's not required). Feel free to go ahead whenever you're ready.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            Checking through the items in DM-10350:

            • DM-10315: does not matter as it only changed defects since 2016
            • DM-10267, DM-10430: changes in middleware
            • DM-10449: should not be essential; previous HSC production were run without it.
            • DM-10451: May be best to include it.

            Here is my proposal: I'll use the w_2017_17 stack with a new, post-DM-10451 obs_subaru, which will basically be w_2017_17 + DM-10315 + DM-10449 + DM-10451.

            John Swinbank what do you think?

            [Edit: post-DM-10451, not post-DM-10129]

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited Checking through the items in DM-10350 : DM-10315 : does not matter as it only changed defects since 2016 DM-10267 , DM-10430 : changes in middleware DM-10449 : should not be essential; previous HSC production were run without it. DM-10451 : May be best to include it. Here is my proposal: I'll use the w_2017_17 stack with a new, post- DM-10451 obs_subaru , which will basically be w_2017_17 + DM-10315 + DM-10449 + DM-10451 . John Swinbank what do you think? [Edit: post-DM-10451, not post-DM-10129]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Two minor changes that should not affect the science pipeline calculations:

            • The "Arcturus" Bright Object Masks will be used (DM-10436)
            • I may change the logging configuration in ctrl_pool (e.g. DM-10430)
            Show
            hchiang2 Hsin-Fang Chiang added a comment - Two minor changes that should not affect the science pipeline calculations: The "Arcturus" Bright Object Masks will be used ( DM-10436 ) I may change the logging configuration in ctrl_pool (e.g. DM-10430 )
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Main documentation of the reprocessing: https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing
            Additional notes on the pipelines: https://confluence.lsstcorp.org/display/~hchiang2/S17B+Output+dataset+types+of+pipe_drivers+tasks+for+HSC

            John Swinbank you may want to wait for the review of DM-10129; please put this to Reviewed when the DRP team is happy with this version/setup/config and I have the permission to start the full PDR1 reprocessing.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Main documentation of the reprocessing: https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing Additional notes on the pipelines: https://confluence.lsstcorp.org/display/~hchiang2/S17B+Output+dataset+types+of+pipe_drivers+tasks+for+HSC John Swinbank you may want to wait for the review of DM-10129 ; please put this to Reviewed when the DRP team is happy with this version/setup/config and I have the permission to start the full PDR1 reprocessing.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            w_2017_17 became available today, and will be used in DM-10129.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - w_2017_17 became available today, and will be used in DM-10129 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The weekly w_2017_17 (~24-Apr-2017) will have all the important HSC fixes.

            Steps/configs: (also noted on https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing )

            Unless otherwise noted, the HSC default config in the stack is used (task defaults + obs_subaru overrides).
            That implies the PS1 reference catalog "ps1_pv3_3pi_20170110" in the LSST format (HTM indexed) is used (/datasets/refcats/htm/ps1_pv3_3pi_20170110/).
            The calibration dataset is the 20170105 version from Paul Price; the calibration repo is located at /datasets/hsc/calib/20170105 from (DM-9978)

            1. makeSkyMap.py
            2. singleFrameDriver.py Ignore ccd=9 which has bad amps and results not trustworthy even if processCcd passes
            3. mosaic.py Not tagged by the weekly; use master on 24-Apr-2017 to go with w_2017_17
            4. coaddDriver.py Make config.assembleCoadd.subregionSize small enough so a full stack of images can fit into memory at once; a trade-off between memory and i/o but doesn't matter scientifically, as the pixels are independent. In other words, config.assembleCoadd.subregionSize may not be the same for all processing, and may be tested and modified as needed.
            5. multiBandDriver.py
            Show
            hchiang2 Hsin-Fang Chiang added a comment - The weekly w_2017_17 (~24-Apr-2017) will have all the important HSC fixes. Steps/configs: (also noted on https://confluence.lsstcorp.org/display/DM/S17B+HSC+PDR1+reprocessing ) Unless otherwise noted, the HSC default config in the stack is used (task defaults + obs_subaru overrides). That implies the PS1 reference catalog "ps1_pv3_3pi_20170110" in the LSST format (HTM indexed) is used (/datasets/refcats/htm/ps1_pv3_3pi_20170110/). The calibration dataset is the 20170105 version from Paul Price; the calibration repo is located at /datasets/hsc/calib/20170105 from ( DM-9978 ) makeSkyMap.py singleFrameDriver.py Ignore ccd=9 which has bad amps and results not trustworthy even if processCcd passes mosaic.py Not tagged by the weekly; use master on 24-Apr-2017 to go with w_2017_17 coaddDriver.py Make config.assembleCoadd.subregionSize small enough so a full stack of images can fit into memory at once; a trade-off between memory and i/o but doesn't matter scientifically, as the pixels are independent. In other words, config.assembleCoadd.subregionSize may not be the same for all processing, and may be tested and modified as needed. multiBandDriver.py

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                John Swinbank
                Watchers:
                Hsin-Fang Chiang, John Swinbank
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel