Details
-
Type:
RFC
-
Status: Implemented
-
Resolution: Done
-
Component/s: DM
-
Labels:None
-
Location:On this issue
Description
Edit: I'm trying to make it more clear what I consider in scope for the "base ISR."
A milestone for this cycle is to deliver ISR to correct for all expected effects excluding those that involve pixel boundary distortions: brighter fatter, tree rings, edge rolloff, etc. The rationale is that we are still learning about the more subtle effects, but we need a basic ISR that will allow processing all data up to current levels. Below I outline three categories of effects.
1. Effects in scope for "Base ISR" – I have put the things in bold that I think we need to spend some effort on.
2. Effects that part of ISR, but out of scope for "Base ISR"
3. Effects that are not part of ISR
Effects in scope for "Base ISR"
- Non-linearity correction
- Intra-chip cross-talk correction
- Basic defect and saturation interpolation
- Dark current correction
- Full frame bias correction
- Overscan correction
- Flatfielding
- Illumination correction
- Amp assembly
- Fringe correction
Effects in ISR, but out of scope for "Base ISR"
- Brighter/Fatter
- Treerings
- Edge and midline rolloff
- Pixel response non-uniformity
- Hysteresis effects from bright stars
- Charge transfer efficiency corrections
- Inter-chip crosstalk
- Advance defect and saturation interpolation (Gaussian processes?)
Aspects that are out of scope for ISR
- Cosmic ray mitigation
- Persistence – My hope is this is handled by a higher level logging/orchestration framework
- QA – We will need to instrument our tasks, but I think that's a different effort
Attachments
Issue Links
- is triggering
-
DM-3878 Implement all ISR corrections for LSST
- Done
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
A milestone for this cycle is to deliver ISR to correct for all expected effects excluding those that involve pixel boundary distortions: brighter fatter, tree rings, edge rolloff, etc. The rationale is that we are still learning about the more subtle effects, but we need a basic ISR that will allow processing all data up to current levels.
What's missing? We do not currently have a working linearity correction. I plan to implement a simple one that will use the information in the ampInfo objects. There is a crosstalk correction that can possibly be ported from obs_subaru, but none exists in ip_isr at the mooment. I plan on implementing an intra-chip crosstalk solution. We will need something better, but probably not in the short term. I'm filing this RFC to make sure nothing else is missing. Current available corrections are: * Dark current correction * Full frame bias correction * Overscan correction * Flatfielding * Illumination correction * Amp assembly * Fringe correction Are there any others I should include? |
A milestone for this cycle is to deliver ISR to correct for all expected effects excluding those that involve pixel boundary distortions: brighter fatter, tree rings, edge rolloff, etc. The rationale is that we are still learning about the more subtle effects, but we need a basic ISR that will allow processing all data up to current levels.
*What's missing?* We do not currently have a working linearity correction. I plan to implement a simple one that will use the information in the ampInfo objects. There is a crosstalk correction that can possibly be ported from obs_subaru, but none exists in ip_isr at the mooment. I plan on implementing an intra-chip crosstalk solution. We will need something better, but probably not in the short term. *What exists now.* I'm filing this RFC to make sure nothing else is missing. Current available corrections are: * Dark current correction * Full frame bias correction * Overscan correction * Flatfielding * Illumination correction * Amp assembly * Fringe correction Are there any others I should include? |
I assume that you include the code to generate the darks/biases/flats? There's code that Paul wrote for HSC that could be ported (I don't know if it's good enough).
Should the list include defect interpolation? Single or double bad columns are currently OK, I think, but we do need to work on interpolating over larger defects (I'd try Gaussian processes, which is the natural generalisation of what we have now). But this is probably not ISR (however, it will help with astrometric registration which UW cares about).
I thought we had non-linearity corrections on the LSST side (I know HSC does). They belong on the list.
I wrote code to measure crosstalk coefficients (within a chip) for HSC using the saturation trails; NAOJ wrote another version using CRs based on a paper by Yagi but I don't know that it's any better. DECam has crosstalk coefficients between chips that are as large as ones within a chip, but I see that you're ruling this out of scope. However, I think we should at least ask what middleware will be needed.
How many fringe components are we able to handle?
What sort of illumination corrections are you planning? Flat fielding needs to leave the sky flat.
Are we calling saturation ISR? How about CTE? Cosmic Rays?
We should be able to handle N fringe components in the low-level algorithmic code, but are currently limited to 1 by the high-level I/O code (e.g., we have no I/O for data cubes) and by the fact that we don't yet have code to calculate the multiple components (I think the pieces are there, e.g., ImagePca, but they haven't been put into the fringe construction code).
Things I see in HSC ISR that we may want on the LSST side (and not mentioned by RHL):
- Brighter-fatter correction (plus the code to compute the coefficients, which is not yet in HSC).
- Vignetting polygon.
- Thumbnails of the flat-fielded images; these are potentially useful for debugging.
- QA metrics.
- Set a rough zero-point, which is mildly helpful for CModel.
Paul Price
added a comment - We should be able to handle N fringe components in the low-level algorithmic code, but are currently limited to 1 by the high-level I/O code (e.g., we have no I/O for data cubes) and by the fact that we don't yet have code to calculate the multiple components (I think the pieces are there, e.g., ImagePca, but they haven't been put into the fringe construction code).
Things I see in HSC ISR that we may want on the LSST side (and not mentioned by RHL):
Brighter-fatter correction (plus the code to compute the coefficients, which is not yet in HSC).
Vignetting polygon.
Thumbnails of the flat-fielded images; these are potentially useful for debugging.
QA metrics.
Set a rough zero-point, which is mildly helpful for CModel.
Michael Wood-Vasey I left of dealing with hysteresis effects like that as out of scope. It seems like dealing with those effects could be handled by custom ISR tasks.
Robert Lupton I should have been more clear. This is just for application of the master frames. Not to build artificial silos, but the effort to write that code is technically located at Princeton and I've been assuming that John S. would schedule that work.
Simple defect and saturation interpolation is in ISR right now. We can take on improving it, but I think it currently is at a level that is acceptable in the short term.
Regarding non-linearity corrections, we have something, but it has not been used in ages and should really be reimplemented (I don't think it's too hard).
For crosstalk, I'm planning on assuming that they are measured by another pipeline and using the obs_subaru version to get a handle on the application. I'm very happy to take notes on what we'd need to do inter-chip corrections, but as you say I think it's out of scope for this particular cycle.
How many fringe components are we able to handle?
I'll have to look into this, but we can talk about it. Currently it's a port of Paul's code from HSC.
What sort of illumination corrections are you planning? Flat fielding needs to leave the sky flat.
I'm interested currently that the capability of applying these corrections exist. We will certainly need to think about the details of implementation.
Are we calling saturation ISR? How about CTE? Cosmic Rays?
Currently saturation is ISR, cosmic rays are not, and we don't treat CTE. I was planning on putting CTE in the same bin as the hysteresis effects some sensors exhibit. That is, make sure the primitives exist, but leave implementation to the specific instruments.
Hopefully that all makes some sense, at least in the short/medium term.
Things I see in HSC ISR that we may want on the LSST side (and not mentioned by RHL):
- Brighter-fatter correction (plus the code to compute the coefficients, which is not yet in HSC).
- Vignetting polygon.
- Thumbnails of the flat-fielded images; these are potentially useful for debugging.
- QA metrics.
- Set a rough zero-point, which is mildly helpful for CModel.
Thanks for the suggestions. These all sound reasonable except that I have explicitly left brighter-fatter off. It appears later in the schedule. If we can get it done now, that would be good, but unless there is a compelling reason to change the schedule the other things will take priority.
I am almost finished porting over the HSC implementation of brighter fatter. I should have it out for review tomorrow. However, as Paul said, it does not calculate the coefficients it only applies them.
Lauren has also already ported over the vignetting polygon stuff.
| Description |
A milestone for this cycle is to deliver ISR to correct for all expected effects excluding those that involve pixel boundary distortions: brighter fatter, tree rings, edge rolloff, etc. The rationale is that we are still learning about the more subtle effects, but we need a basic ISR that will allow processing all data up to current levels.
*What's missing?* We do not currently have a working linearity correction. I plan to implement a simple one that will use the information in the ampInfo objects. There is a crosstalk correction that can possibly be ported from obs_subaru, but none exists in ip_isr at the mooment. I plan on implementing an intra-chip crosstalk solution. We will need something better, but probably not in the short term. *What exists now.* I'm filing this RFC to make sure nothing else is missing. Current available corrections are: * Dark current correction * Full frame bias correction * Overscan correction * Flatfielding * Illumination correction * Amp assembly * Fringe correction Are there any others I should include? |
Edit: I'm trying to make it more clear what I consider in scope for the "base ISR."
A milestone for this cycle is to deliver ISR to correct for all expected effects excluding those that involve pixel boundary distortions: brighter fatter, tree rings, edge rolloff, etc. The rationale is that we are still learning about the more subtle effects, but we need a basic ISR that will allow processing all data up to current levels. Below I outline three categories of effects. 1. Effects in scope for "Base ISR" -- I have put the things in bold that I think we need to spend some effort on. 2. Effects that part of ISR, but out of scope for "Base ISR" 3. Effects that are not part of ISR *Effects in scope for "Base ISR"* * *Non-linearity correction* * *Intra-chip cross-talk correction* * Basic defect and saturation interpolation * Dark current correction * Full frame bias correction * Overscan correction * Flatfielding * *Illumination correction* * Amp assembly * *Fringe correction* *Effects in ISR, but out of scope for "Base ISR"* * Brighter/Fatter * Treerings * Edge and midline rolloff * Pixel response non-uniformity * Hysteresis effects from bright stars * Charge transfer efficiency corrections * Inter-chip crosstalk * Advance defect and saturation interpolation (Gaussian processes?) *Aspects that are out of scope for ISR* * Cosmic ray mitigation * Persistence -- My hope is this is handled by a higher level logging/orchestration framework * QA -- We will need to instrument our tasks, but I think that's a different effort |
Thanks to all for the comments. I've tried to tidy things up a bit. If there are no more comments by Monday, I'll mark this as adopted.
| Resolution | Done [ 10000 ] | |
| Status | Proposed [ 10805 ] | Adopted [ 10806 ] |
Did this decision trigger any work? There are no tickets listed on this RFC as resulting from the adoption.
I'll reiterate my comment from last April. Is there any work associated with this RFC?
DM-3878 is the implementation epic, I believe.
Yes, sorry. DM-3878 is the relevant epic.
| Status | Adopted [ 10806 ] | Implemented [ 11105 ] |
Is persistence beyond scope?
E.g., when the subsequent images still has left over effects from the 6th magnitude star that was on that CCD in a previous 15-sec exposure on a >2-m telescope.