# Implement global sky subtraction

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
10
• Team:
External

#### Description

Masayuki Tanaka and Michitaro Koike (NAOJ) have put together a sky subtraction code (based on the HSC pipeline) that performs better for large objects than what we are currently using. Their approach creates sky frames from many observations and subtracts these from the data, with additional large-scale subtraction to deal with any small residuals. I need to:
1. Understand the details of the approach taken.
2. RFC this addition to the processing.
3. Implement the code on the LSST side.

Their code is on the koike/global-sky-subtraction branch of hscPipe.

#### Attachments

1. background.png
380 kB
2. sky.png
532 kB
3. visit35850.png
434 kB

#### Activity

Hide
Paul Price added a comment -

Thanks for volunteering to review, Pim Schellart [X].

I'll try to provide some new demonstration images over the weekend, but I'm off to bed now and on a plane tomorrow.

Show
Paul Price added a comment - Thanks for volunteering to review, Pim Schellart [X] . I'll try to provide some new demonstration images over the weekend, but I'm off to bed now and on a plane tomorrow.
Hide
Pim Schellart [X] (Inactive) added a comment -

See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too.

Show
Pim Schellart [X] (Inactive) added a comment - See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too.
Hide
Paul Price added a comment -

Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow.

Show
Paul Price added a comment - Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow.
Hide
Paul Price added a comment -

sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for.

##### Instructions

The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line:

 constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0 

Application of the sky correction is done using skyCorrection.py, which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr):

 skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 

In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True, e.g.:

 coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True 

Show
Paul Price added a comment - sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for. Instructions The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line: constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0 Application of the sky correction is done using skyCorrection.py , which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr ): skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True , e.g.: coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
Hide
Paul Price added a comment -

Thank you all!

Merged to master.

Show
Paul Price added a comment - Thank you all! Merged to master.

#### People

Assignee:
Paul Price
Reporter:
Paul Price
Reviewers:
Pim Schellart [X] (Inactive)
Watchers:
Hsin-Fang Chiang, Paul Price, Pim Schellart [X] (Inactive)