Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17229

Telecon (1/16/19) with SAWG (LSST DESC) to discuss priority of sensor effects

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      A. Plazas, M. Fisher-Levine (DRP), P. Astier, P. Antilogus, A. Nomerotski (SAWG) will have a Zoom telecon on 01/16/19 regarding the status and priority of known sensor effects (e.g., https://confluence.slac.stanford.edu/pages/viewpage.action?spaceKey=LSSTDESC&title=Known+Sensor+Effects+and+Anomalies). Some questions to guide the discussion: 

      • What are the effects that we know or suspect will have an impact on LSST science (taking into account the science goals and requirements)?
      • For those effects, are there any algorithms to correct for them to acceptable levels or is this still a topic of investigation?
      • If there are, should they be implemented as part of the DM pipelines? What inputs would be needed?

       

       

        Attachments

          Issue Links

            Activity

            Hide
            swinbank John Swinbank added a comment -

            This is a model ticket. Thanks Andrés Alejandro Plazas Malagón!

            Show
            swinbank John Swinbank added a comment - This is a model ticket. Thanks Andrés Alejandro Plazas Malagón !
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Comment by Pierre Antilogus (from an email communication previous to the meeting):

            1) what effect do we see ===> this is the list: https://confluence.slac.stanford.edu/pages/viewpage.action?spaceKey=LSSTDESC&title=Known+Sensor+Effects+and+Anomalies)

            2) what effect matter ==> we could for most of them put an answer even if this is not enough quantified yet for most of them for such key question

            3) do we know how to correct them? what data are needed for that? can we estimate left over systematic once corrected  ==> clearly lots of work left to do

            I think we do have lots of information , but we should agree at this stage how ( which form and where ) we proceed from there. Let's discuss this over a Telecon (purpose of this ticket)

             

             

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Comment by Pierre Antilogus (from an email communication previous to the meeting): 1) what effect do we see ===> this is the list: https://confluence.slac.stanford.edu/pages/viewpage.action?spaceKey=LSSTDESC&title=Known+Sensor+Effects+and+Anomalies ) 2) what effect matter ==> we could for most of them put an answer even if this is not enough quantified yet for most of them for such key question 3) do we know how to correct them? what data are needed for that? can we estimate left over systematic once corrected  ==> clearly lots of work left to do I think we do have lots of information , but we should agree at this stage how ( which form and where ) we proceed from there. Let's discuss this over a Telecon (purpose of this ticket)    
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Comment by Pierre Astier (from an email communication previous to the meeting): 

            For static distortions, the scheme implemented in DES seems
            to work, although there are systematic PSF size residuals that display a tree ring structure, although the PSF is
            computed (I guess) in the transformed coordinate system (This was shown by Pierre-François Léget at some
            PSF or Piff meeting, I can find the slides). I am puzzled by those. One difficulty though is that the
            WCS scheme being developed for LSST is not delivered yet (AFAIK), and we hence cannot start to exercise.
            Andrei and Co have produced some work about the impact of tree rings on shape measurements, and
            there is more work planed (dixit Heyhun ?) , so we might get some new insight here. Personally
            I am more concerned about sensor edge effects than about tree rings (at the scale they have on LSST sensors),
            but I might be wrong. At least some sort of assessment and planning exists there, and it is in the hands of
            BNL.

            Regarding BF, I am quite confident that we still do not know what algorithm LSST will run. One point
            that could probably gather a good consensus is that the current state of the art is at best marginal w.r.t.
            LSST objectives, although a formal study would be welcome. I think that now, all considered observing
            plans include the "rotational dithers", in order to average out the anisotropy of the (left over) effect, which
            reduces the load on the quality of the anisotropic correction, if the size is OK, and enough images at play.
            In SAWG, we are trying to come up with some serious test plan (calibration data->model,
            application of the model to test data), but nothing convincing (and easy enough to implement)
            has shown up. People informally involved are the usual suspects: Craig, Andy, and the Paris folks.

            Having a telecon is probably a good idea.

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Comment by Pierre Astier (from an email communication previous to the meeting):  For static distortions, the scheme implemented in DES seems to work, although there are systematic PSF size residuals that display a tree ring structure, although the PSF is computed (I guess) in the transformed coordinate system (This was shown by Pierre-François Léget at some PSF or Piff meeting, I can find the slides). I am puzzled by those. One difficulty though is that the WCS scheme being developed for LSST is not delivered yet (AFAIK), and we hence cannot start to exercise. Andrei and Co have produced some work about the impact of tree rings on shape measurements, and there is more work planed (dixit Heyhun ?) , so we might get some new insight here. Personally I am more concerned about sensor edge effects than about tree rings (at the scale they have on LSST sensors), but I might be wrong. At least some sort of assessment and planning exists there, and it is in the hands of BNL. Regarding BF, I am quite confident that we still do not know what algorithm LSST will run. One point that could probably gather a good consensus is that the current state of the art is at best marginal w.r.t. LSST objectives, although a formal study would be welcome. I think that now, all considered observing plans include the "rotational dithers", in order to average out the anisotropy of the (left over) effect, which reduces the load on the quality of the anisotropic correction, if the size is OK, and enough images at play. In SAWG, we are trying to come up with some serious test plan (calibration data->model, application of the model to test data), but nothing convincing (and easy enough to implement) has shown up. People informally involved are the usual suspects: Craig, Andy, and the Paris folks. Having a telecon is probably a good idea.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited

            Notes from the meeting.

            • Overall, we went over the list “Known Sensor Effects” Pierre Antilogus explained and illustrated each effect. We also discussed the relationship between
            • Before going into the list, some comments:
              • The list is not complete as more R&D is being done. There are other effects that are not there because they are part of the standard ISR steps. However, even some these “standard” or “regular “steps need to be looked at in detail for LSST devices. For example, bias subtraction does not seem stable in some case.
            • SAWG:
              • Help with identification effects DM should care about (raising flags)
              • Help with validation. However, need to define what this means, which metrics to use (impacts on photometry, astrometry, and shapes?)
            • Tools (data):
              • Lab data: in some cases, easier to do things in the lab.
              • Simulations: close the loop (measured vs truth)
              • Data from the mountain
              • Other datasets
            • List of Known Sensor Effects:
              • BF: still more work to do but progress going on (e.g., by Pierre Astier, Any Rasmussen, etc.). Current correction in DM is Coulton correction; corrects for 90% of the effect using information from flats (i.e., missing information from focused light).
              • Static sensor effects
                • Tree rings: small for LSST, but still need to check. DES method?
                • Edge distortion: DES method?
                • Spider legs: Not clear what they are, but they don’t show up anymore. But they might re-appear.
                • Bamboo: high-spatial frequency change in size (geometric )pixel size effect, not that clear what to do.
                • Pixel size effects: more generally, need to investigate how measurements algorithms perform when the pixel grid is not regular.
              • Flat distortion
                • Tearing (e2v): there is a voltage/clock configuration to solve the problem, but it is still under investigation how safely it can be implement (how low can you go in voltage). But it will be fixed.
                • Dipole: collection level problem, it was a mistake in the configuration the sensors, but it is something that can be fixed/removed.
              • Sensor defects (bright defects, dead pixels)
                • General comment:
                  • Nothing out of the ordinary regarding usual bright/dead pixels.
                  • However, it has been observed that an offset in the bias level is present due to defect/bright pixel/column. Could be corrected using the overscan, but is still under investigation (see Yusuke Utsumi poster at ISPA 2018)
              • CTE and trapping issues:
                • Some devices (especially ITL) still present large CTI. A correction might be needed (à la Massey et. al, MNRAS 439.1, Anderson and Bedin PASP 895 for CTI in ACS/WFC). Still open for discussion at this stage.
              • Fringes:
                • We have on-sky and lab data
                • PCA for fringe correction
              • Acquisition related issues:
                • ADU favoritism: In histogram from flats, some ADU values deviate from Poissonian distribution (“holes” or “dips” can be seen in the distribution). Need to investigate more about its impact on observables.
                • Pixel to pixel memory: temporal type of crosstalk; change of baseline. Small crosstalk at the level of electronics. Time component has not been studied.
                • Jitter: timing of acquisition has some jitter. Can affect gain, but likely not too important.
            • How can we move forward?
              • Current DM code:
                • Calibration Products Production and Instrument Signature Removal in lsst-dm (github)
                • What do we need to include? Do are the priorities? Do we have algorithms that we should include in the DM pipeline?
              • Validation (using lsp at NCSA)
              • R&D
            • Meeting next week (01/23/19), same time (3 p.m Paris; 9 a.m EST), same zoom channel (SAWG meeting channel)
            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - - edited Notes from the meeting. Overall, we went over the list “Known Sensor Effects” Pierre Antilogus explained and illustrated each effect. We also discussed the relationship between Before going into the list, some comments: The list is not complete as more R&D is being done. There are other effects that are not there because they are part of the standard ISR steps. However, even some these “standard” or “regular “steps need to be looked at in detail for LSST devices. For example, bias subtraction does not seem stable in some case. SAWG: Help with identification effects DM should care about (raising flags) Help with validation. However, need to define what this means, which metrics to use (impacts on photometry, astrometry, and shapes?) Tools (data): Lab data: in some cases, easier to do things in the lab. Simulations: close the loop (measured vs truth) Data from the mountain Other datasets List of Known Sensor Effects: BF: still more work to do but progress going on (e.g., by Pierre Astier, Any Rasmussen, etc.). Current correction in DM is Coulton correction; corrects for 90% of the effect using information from flats (i.e., missing information from focused light). Static sensor effects Tree rings: small for LSST, but still need to check. DES method? Edge distortion: DES method? Spider legs: Not clear what they are, but they don’t show up anymore. But they might re-appear. Bamboo: high-spatial frequency change in size (geometric )pixel size effect, not that clear what to do. Pixel size effects: more generally, need to investigate how measurements algorithms perform when the pixel grid is not regular. Flat distortion Tearing (e2v): there is a voltage/clock configuration to solve the problem, but it is still under investigation how safely it can be implement (how low can you go in voltage). But it will be fixed. Dipole: collection level problem, it was a mistake in the configuration the sensors, but it is something that can be fixed/removed. Sensor defects (bright defects, dead pixels) General comment: Nothing out of the ordinary regarding usual bright/dead pixels. However, it has been observed that an offset in the bias level is present due to defect/bright pixel/column. Could be corrected using the overscan, but is still under investigation (see Yusuke Utsumi poster at ISPA 2018) CTE and trapping issues: Some devices (especially ITL) still present large CTI. A correction might be needed (à la Massey et. al, MNRAS 439.1, Anderson and Bedin PASP 895 for CTI in ACS/WFC). Still open for discussion at this stage. Fringes: We have on-sky and lab data PCA for fringe correction Acquisition related issues: ADU favoritism: In histogram from flats, some ADU values deviate from Poissonian distribution (“holes” or “dips” can be seen in the distribution). Need to investigate more about its impact on observables. Pixel to pixel memory: temporal type of crosstalk; change of baseline. Small crosstalk at the level of electronics. Time component has not been studied. Jitter: timing of acquisition has some jitter. Can affect gain, but likely not too important. How can we move forward? Current DM code: Calibration Products Production and Instrument Signature Removal in lsst-dm (github) What do we need to include? Do are the priorities? Do we have algorithms that we should include in the DM pipeline? Validation (using lsp at NCSA) R&D Meeting next week (01/23/19), same time (3 p.m Paris; 9 a.m EST), same zoom channel (SAWG meeting channel)
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Meeting done. Notes are in the comments. We will have another meeting next week. 

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Meeting done. Notes are in the comments. We will have another meeting next week. 
            Hide
            swinbank John Swinbank added a comment -

            Thanks for the awesome notes, Andrés Alejandro Plazas Malagón!

            Show
            swinbank John Swinbank added a comment - Thanks for the awesome notes, Andrés Alejandro Plazas Malagón !

              People

              Assignee:
              plazas Andrés Alejandro Plazas Malagón
              Reporter:
              plazas Andrés Alejandro Plazas Malagón
              Watchers:
              Andrés Alejandro Plazas Malagón, John Swinbank, Merlin Fisher-Levine
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.