Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9147

Implement global sky subtraction

    Details

    • Story Points:
      10
    • Team:
      External

      Description

      Masayuki Tanaka and Michitaro Koike (NAOJ) have put together a sky subtraction code (based on the HSC pipeline) that performs better for large objects than what we are currently using. Their approach creates sky frames from many observations and subtracts these from the data, with additional large-scale subtraction to deal with any small residuals. I need to:
      1. Understand the details of the approach taken.
      2. RFC this addition to the processing.
      3. Implement the code on the LSST side.

      Their code is on the koike/global-sky-subtraction branch of hscPipe.

        Attachments

        1. background.png
          background.png
          380 kB
        2. sky.png
          sky.png
          532 kB
        3. visit35850.png
          visit35850.png
          434 kB

          Issue Links

            Activity

            Hide
            price Paul Price added a comment -

            Thanks for volunteering to review, Pim Schellart [X].

            I'll try to provide some new demonstration images over the weekend, but I'm off to bed now and on a plane tomorrow.

            Show
            price Paul Price added a comment - Thanks for volunteering to review, Pim Schellart [X] . I'll try to provide some new demonstration images over the weekend, but I'm off to bed now and on a plane tomorrow.
            Hide
            pschella Pim Schellart [X] (Inactive) added a comment -

            See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too.

            Show
            pschella Pim Schellart [X] (Inactive) added a comment - See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too.
            Hide
            price Paul Price added a comment -

            Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow.

            Show
            price Paul Price added a comment - Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow.
            Hide
            price Paul Price added a comment -

            sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for.

            Instructions

            The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line:

            constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0
            

            Application of the sky correction is done using skyCorrection.py, which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr):

            skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2
            

            In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True, e.g.:

            coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
            

            Show
            price Paul Price added a comment - sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for. Instructions The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line: constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0 Application of the sky correction is done using skyCorrection.py , which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr ): skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True , e.g.: coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
            Hide
            price Paul Price added a comment -

            Thank you all!

            Merged to master.

            Show
            price Paul Price added a comment - Thank you all! Merged to master.

              People

              • Assignee:
                price Paul Price
                Reporter:
                price Paul Price
                Reviewers:
                Pim Schellart [X] (Inactive)
                Watchers:
                Hsin-Fang Chiang, Paul Price, Pim Schellart [X] (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel