Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33629

Generate updated HSC sky frames

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 18
    • Data Release Production
    • No

    Description

      This ticket captures the effort involved in construction of updated HSC sky frames. Sky frames are constructed for all filters currently on disk at NCSA, including the newer HSC-I2 and HSC-R2 filters.

      As a precursor to DM-32378, updated sky frames with the amp offset correction task turned off ('vanilla' sky frames) are first required. Once a test run (likely an RC2 run) with these sky frames in place has taken place, a future ticket will generate updated sky frames with amp offset corrections turned on.

      Following prior DM efforts in the construction of sky frames, and also at the advice of NAOJ, here we construct sky frames using only science exposures with an exposure time of greater than 30 seconds. Further information on the NAOJ procedures can be found at this link.

      To generate sky frames, all raw exposures in all seven filters are split up into exposure blocks per filter. Each per-filter exposure block captures a specific date-range of input exposures, to capture the time-variant nature of sky frames. Exposure blocks were chosen such that each block consists of at least 50 unique exposures with an exposure time of greater than 30 seconds. However, efforts were made not to split up exposures captured within seven days of a prior exposure in the same filter (i.e., on the same run). If such a scenario occurred, these nearby-neighbour exposures were included into the prior exposure block. Furthermore, if the final exposure block did not meet the >50 criteria, it too was merged into the preceeding exposure block. Details of each exposure block are provided in the comments below.

      Example full focal plane plots comparing this round of sky frames (gen3/vanilla) to the prior round of sky frames (gen2/20200115) are also provided in the comments below. Finally, an example run of the DRP.yaml pipeline tasks skyCorr, makeWarp and assembleCoadd using the old gen2 sky frames and the new gen3 sky frames are also provided below. The new sky frames appear to return a flatter and more even sky in the coadd, however, this should be tested at-scale in an upcoming RC2 run.

      Further analysis plots and movies showing the time-variant nature of sky frames can be found at this link. The full Jupyter notebook used for this study is attached to this ticket. Jobs were submitted using the BPS batch processing system, using the $CP_PIPE_DIR/pipelines/cpSky.yaml pipeline. For reference, an example BPS submit script used here is provided below:

      campaign: DM-33629
      pipelineYaml: $CP_PIPE_DIR/pipelines/cpSky.yaml
      computeSite: ncsapool
      requestMemory: 2048
      requestMemoryMax: 32768
      memoryMultiplier: 2.0
      numberOfRetries: 5
      payload:
          butlerConfig: /repo/main
          inCollection: HSC/defaults
          output: HSC/calib/DM-33629/HSC-G/20140918
          dataQuery: "instrument='HSC' AND exposure.observation_type='science' AND detector!=9 AND exposure.exposure_time>30.0 AND exposure.target_name='SSP-Wide' AND physical_filter='HSC-G' AND exposure.day_obs>=20140918 AND exposure.day_obs<=20141001"
          extraQgraphOptions: "--instrument lsst.obs.subaru.HyperSuprimeCam -c isr:doAmpOffset=False"
          extraRunQuantumOptions: ""
      

      All gen3 vanilla sky frames have been certified into the HSC/calib/DM-33629 calibration collection.

      Attachments

        1. gen2-20200115-sky-HSC-G.mp4
          1.36 MB
        2. gen2-20200115-sky-HSC-I.mp4
          296 kB
        3. gen2-20200115-sky-HSC-I2.mp4
          1.00 MB
        4. gen2-20200115-sky-HSC-R.mp4
          344 kB
        5. gen2-20200115-sky-HSC-R2.mp4
          688 kB
        6. gen2-20200115-sky-HSC-Y.mp4
          890 kB
        7. gen2-20200115-sky-HSC-Z.mp4
          1.21 MB
        8. gen3-vanilla-sky-HSC-G.mp4
          1.44 MB
        9. gen3-vanilla-sky-HSC-G-1.mp4
          1.44 MB
        10. gen3-vanilla-sky-HSC-I.mp4
          284 kB
        11. gen3-vanilla-sky-HSC-I2.mp4
          569 kB
        12. gen3-vanilla-sky-HSC-R.mp4
          557 kB
        13. gen3-vanilla-sky-HSC-R2.mp4
          465 kB
        14. gen3-vanilla-sky-HSC-Y.mp4
          1.01 MB
        15. gen3-vanilla-sky-HSC-Z.mp4
          1.07 MB
        16. HSC_CCDPosition_20210827.png
          HSC_CCDPosition_20210827.png
          211 kB
        17. skyFramesVanilla.ipynb
          7.87 MB
        18. sky-HSC-G-20141111-gen2.png
          sky-HSC-G-20141111-gen2.png
          721 kB
        19. sky-HSC-G-20141118-gen3.png
          sky-HSC-G-20141118-gen3.png
          678 kB
        20. sky-HSC-I-20140328-gen2.png
          sky-HSC-I-20140328-gen2.png
          685 kB
        21. sky-HSC-I-20140328-gen3.png
          sky-HSC-I-20140328-gen3.png
          552 kB
        22. sky-HSC-Y-20160208-gen2.png
          sky-HSC-Y-20160208-gen2.png
          479 kB
        23. sky-HSC-Y-20160212-gen3.png
          sky-HSC-Y-20160212-gen3.png
          493 kB
        24. t9813p24i_diff.png
          t9813p24i_diff.png
          1.04 MB
        25. t9813p24i_gen2.png
          t9813p24i_gen2.png
          2.33 MB
        26. t9813p24i_gen3.png
          t9813p24i_gen3.png
          2.33 MB

        Issue Links

          Activity

            lskelvin Lee Kelvin added a comment - - edited

            Snapshot comparison: HSC-G gen2 sky frame (20141111, left) vs gen3 sky frame (20141118, right).

            The new sky frames seem to exhibit fewer artefacts, such as detectors 86, 88 and 30¹. This is likely due in part to the higher number count limit utilised here per exposure block (gen2=20, gen3=50).

            ¹ for reference:

            lskelvin Lee Kelvin added a comment - - edited Snapshot comparison: HSC-G gen2 sky frame (20141111, left) vs gen3 sky frame (20141118, right). The new sky frames seem to exhibit fewer artefacts, such as detectors 86, 88 and 30¹. This is likely due in part to the higher number count limit utilised here per exposure block (gen2=20, gen3=50). ¹ for reference:
            lskelvin Lee Kelvin added a comment - - edited

            Snapshot comparison: HSC-I gen2 sky frame (20140328, left) vs gen3 sky frame (20140328, right).

            The prior HSC-I sky frames exhibit unusual low/high detector values in detectors 102/100 (respectively). The gen3 sky frames do not appear to exhibit such features.

            lskelvin Lee Kelvin added a comment - - edited Snapshot comparison: HSC-I gen2 sky frame (20140328, left) vs gen3 sky frame (20140328, right). The prior HSC-I sky frames exhibit unusual low/high detector values in detectors 102/100 (respectively). The gen3 sky frames do not appear to exhibit such features.
            lskelvin Lee Kelvin added a comment -

            Snapshot comparison: HSC-Y gen2 sky frame (20160208, left) vs gen3 sky frame (20160212, right).

            The new sky frame procedure (gen3, right) appears less impacted by the presence of dead amps, or partially dead amps.

            lskelvin Lee Kelvin added a comment - Snapshot comparison: HSC-Y gen2 sky frame (20160208, left) vs gen3 sky frame (20160212, right). The new sky frame procedure (gen3, right) appears less impacted by the presence of dead amps, or partially dead amps.
            lskelvin Lee Kelvin added a comment - - edited

            As a test, calexps from the HSC/runs/RC2/w_2021_50/DM-32987 RUN collection were used to reduce an example patch: tract 9813, patch 24 in HSC-I. The pipeline ran was: $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#skyCorr,makeWarp,assembleCoadd. The plots below show the tract reduced using the current/gen2 sky frames, and reduced using the new (this ticket) gen3 sky frames, as well as a difference image.

            The reduction using the new sky frame procedure appears to return a smoother and more uniform background sky, producing fewer excessive negative troughs, and seemingly performing better at not oversubtracting flux surrounding bright sources. A future test RC2 run using these new data should be used to thoroughly test this however.

            lskelvin Lee Kelvin added a comment - - edited As a test, calexps from the HSC/runs/RC2/w_2021_50/ DM-32987 RUN collection were used to reduce an example patch: tract 9813, patch 24 in HSC-I. The pipeline ran was: $DRP_PIPE_DIR/pipelines/HSC/DRP-RC2.yaml#skyCorr,makeWarp,assembleCoadd . The plots below show the tract reduced using the current/gen2 sky frames, and reduced using the new (this ticket) gen3 sky frames, as well as a difference image. The reduction using the new sky frame procedure appears to return a smoother and more uniform background sky, producing fewer excessive negative troughs, and seemingly performing better at not oversubtracting flux surrounding bright sources. A future test RC2 run using these new data should be used to thoroughly test this however.
            price Paul Price added a comment -

            Pictures look very convincing, and I love the effort you've gone to document all your decisions here. Great work!

            price Paul Price added a comment - Pictures look very convincing, and I love the effort you've gone to document all your decisions here. Great work!

            People

              lskelvin Lee Kelvin
              lskelvin Lee Kelvin
              Paul Price
              Lee Kelvin, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.