Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: ctrl_pool, obs_base, obs_subaru, pipe_drivers, pipe_tasks
-
Labels:
-
Story Points:10
-
Team:External
Description
Masayuki Tanaka and Michitaro Koike (NAOJ) have put together a sky subtraction code (based on the HSC pipeline) that performs better for large objects than what we are currently using. Their approach creates sky frames from many observations and subtracts these from the data, with additional large-scale subtraction to deal with any small residuals. I need to:
1. Understand the details of the approach taken.
2. RFC this addition to the processing.
3. Implement the code on the LSST side.
Their code is on the koike/global-sky-subtraction branch of hscPipe.
Attachments
Issue Links
Activity
See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too.
Pim Schellart [X] (Inactive)
added a comment - See PR. Perhaps Hsin-Fang Chiang still wants to look at the multi-node processing steps, but that looks in good shape too. Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow.
Paul Price
added a comment - Thanks Pim! I've gone through your comments to clean up the code, and hope to exercise it tomorrow. sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for.
Instructions
The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line:
constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0
|
Application of the sky correction is done using skyCorrection.py, which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr):
skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2
|
In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True, e.g.:
coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
|
Paul Price
added a comment - sky.png shows an example coadd (Cosmos, r-band, tract=9813, patch=5,5) constructed using the sky correction (left) and the same processed with vanilla hscPipe 5.0-beta4 (right). The dark (negative) halos around bright objects in the old processing (right) are not present, and the wings are much more visible with the sky correction turned on (left). I believe that's the result we were looking for.
Instructions
The sky correction requires a sky frame, which is constructed in the same manner as biases, darks and flats. I'm not sure about the time scale on which the sky frames vary. I don't think they should vary much at all, but since it's cheap I suggest we construct them for every night for which we have ~20 or more (long) SSP_WIDE observations, and then ingest them with a validity range that will allow them to be used for nearby nights that don't have such. We should not include SSP_DEEP_* or SSP_UDEEP_* observations in the sky frame construction, as they don't have much dithering and I fear objects may print through. Here's an example command-line:
constructSky.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-sky --batch-type=slurm --cores=140 --mpiexec="-bind-to socket" --time=0.5 --job sky-r --no-versions --id filter=HSC-R dateObs=2016-06-11 field=SSP_WIDE expTime=150.0
Application of the sky correction is done using skyCorrection.py , which writes a background model that replaces the background previously removed as part of single frame processing and removes an exposure-wide background (1000 pixel-sized super-pixels) and a scaled sky frame. Note that this doesn't modify anything, but only writes a new background model (dataset type skyCorr ):
skyCorrection.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --job skyCorr-r --time 60 --cores 112 --batch-type slurm --mpiexec='-bind-to socket' --id field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2
In order to have the sky correction applied during warping, you need to set doApplySkyCorr=True , e.g.:
coaddDriver.py /tigress/HSC/HSC --calib /tigress/pprice/dm-9147/calibs --rerun price/dm-9147-cosmos --id tract=9813 filter=HSC-R --selectId field=COSMOS^SSP_UDEEP_COSMOS filter=HSC-R^HSC-R2 --job coadd-r --time 100 --cores 16 --batch-type=slurm --mpiexec='-bind-to socket' -c makeCoaddTempExp.doApplySkyCorr=True
Thank you all!
Merged to master.
Paul Price
added a comment - Thank you all!
Merged to master. 
Thanks for volunteering to review, Pim Schellart [X].
I'll try to provide some new demonstration images over the weekend, but I'm off to bed now and on a plane tomorrow.