Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: cp_pipe, pipe_drivers
-
Labels:None
-
Story Points:18
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
This ticket aims to regenerate HSC sky frames with the amp offset correction task turned on.
HSC data are known to exhibit small (~few ADU) amp-to-amp offsets which persist post-ISR correction. The origin of this offset is believed to be a feature of the hardware, requiring an empirical correction (ampOffsetTask) after the fact. This task was implemented in DM-24389.
HSC data undergo a full-focal-plane sky correction prior to coadd-level data processing, producing superior sky solutions on larger scales. This task was initially implemented in DM-9147, and converted to gen3 in DM-22534 (cpSkyTask). Following DM-33629, HSC sky frames were updated using the latest cpPipe implementation, however, amp-offset corrections were left switched off to allow for a direct comparison with the older data processing outputs.
To that end, the calibration sky frames currently in use do not have amp offset corrections applied, therefore requiring regeneration on this ticket. Once tested as part of an RC2 run, these new amp-offset corrected sky frames should replace those currently in use for standard HSC data processing.
Attachments
Issue Links
- is duplicated by
-
- Invalid
Activity
Looks good. OK to merge.
- You actually have until the new shared stack gets built tomorrow to switch the default HSC/calib (if you want more sleep
) - We need to advertise that if someone is processing RC2 with an older stack people need to use the SkyFrames in `HSC/calib/
DM-33629`. I THINK doing {{ -i HSC/calib/DM-33629,HSC/RC2/defaults}} would do it, but Jim Bosch or Nate Lust name can correct me.
We can continue to do QA after it is switched on.
1) The images that I've been dying to see are the full visits with:
- New Vanilla SkyFrames + skyCorr subtracted off vs. doAmpOffset = True +
DM-32378Sky Frames + skyCorr subtracted off.
2) Eli Rykoff promised to make some a before and after photometric residual plots (summed over detector rows to maybe yield enough signal with just RC2 – e.g. 1D ramps vs 2D ramps)
3) I promised to look at the warp diffs to see if they look more suitable for background matching.
That's great, thanks Yusra. Ticket merged just now, branch deleted, and the HSC/calib collection updated!
Will add posts advertising the change on Slack into #dm-hsc-reprocessing and #dm-science-pipelines. Will also look at generating full visit images this week, and attach them to this ticket.
Thanks again!
I did promise this! I will look as soon as the upcoming RC2 processing gets far enough.
In order to switch amp offset correction on by default, two things need to happen:
To update HSC/calib, the old vanilla sky frame calibration collection (HSC/calib/
DM-33629) needs to be removed, and the new sky frame calibration collection (HSC/calib/DM-32378) added. This can be achieved on the command line using the following:butler collection-chain /repo/main --mode remove HSC/calib HSC/calib/DM-33629followed by
butler collection-chain /repo/main --mode prepend HSC/calib HSC/calib/DM-32378Ideally, merging this ticket to the main branch and performing the above chained collection redefinition should occur as close to one-another in time as possible.