Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-683

Move crosstalk measurement code and change data format.

    Details

    • Type: RFC
    • Status: Withdrawn
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      1) measureCrosstalk.py should live in cp_pipe now that it exists, as it's creating a  calibration product, so step 1 is to simply move it there, and move the writeCrosstalkCoeffs method as well, as that really belongs with the measuring code.

      2) Update the data format for the crosstalk coefficients. Current format written is a 4-deep nested dict, with the two outer layers being 100% superfluous. The proposed change to the format will either be to just rip these out, or better, make a python defects class holding the coefficients so that they can be used by ampName (currently they're stored like that, but during the loading the names are discarded and they're thrown into a numpy array, losing the safety that the dict keys provide), which would involve making the requisite changes to the isrTask to enable their use.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Are these implemented in a CrossTalk class that has read/write methods for serialization? Or are they used as python dicts directly?

            Show
            tjenness Tim Jenness added a comment - Are these implemented in a CrossTalk class that has read/write methods for serialization? Or are they used as python dicts directly?
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            At the moment, neither. There's no class, and a dict of dicts of dicts of dicts is written manually in yaml, and when retrieved, the two outer layers are discarded, and the inner two, which give the coefficients by name, have their naming-protection discarded, and the values shoved into a naked numpy array.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - At the moment, neither. There's no class, and a dict of dicts of dicts of dicts is written manually in yaml, and when retrieved, the two outer layers are discarded, and the inner two, which give the coefficients by name, have their naming-protection discarded, and the values shoved into a naked numpy array.
            Hide
            plazas Andrés Alejandro Plazas Malagón added a comment -

            Example:

            /project/shared/auxTel/CALIB/crosstalk/crosstalk-S00-det000.yaml
            

            {'crosstalk': {'auxTel': {'C10': {'C10': 0.0,
                'C11': 0.00041452,
                'C12': 5.3125e-05,
                'C13': 1.0496e-05,
                'C14': -7.5229e-06,
                'C15': -2.0852e-05,
                'C16': -3.0844e-05,
                'C17': -2.7888e-05,
                'C07': -3.1578e-05,
                'C06': -3.738e-05,
                'C05': -3.806e-05,
                'C04': -4.1003e-05,
                'C03': -3.189e-05,
                'C02': -3.4353e-05,
                'C01': -3.5075e-05,
                'C00': -3.8668e-05},
               'C11': {'C10': 0.00029295,
                'C11': 0.0,
                'C12': 0.00039701,
                'C13': 0.0001004,
            

            Show
            plazas Andrés Alejandro Plazas Malagón added a comment - Example: /project/shared/auxTel/CALIB/crosstalk/crosstalk-S00-det000.yaml {'crosstalk': {'auxTel': {'C10': {'C10': 0.0, 'C11': 0.00041452, 'C12': 5.3125e-05, 'C13': 1.0496e-05, 'C14': -7.5229e-06, 'C15': -2.0852e-05, 'C16': -3.0844e-05, 'C17': -2.7888e-05, 'C07': -3.1578e-05, 'C06': -3.738e-05, 'C05': -3.806e-05, 'C04': -4.1003e-05, 'C03': -3.189e-05, 'C02': -3.4353e-05, 'C01': -3.5075e-05, 'C00': -3.8668e-05}, 'C11': {'C10': 0.00029295, 'C11': 0.0, 'C12': 0.00039701, 'C13': 0.0001004,
            Hide
            czw Christopher Waters added a comment -

            I am 100% onboard with #1.  I think that's the only thing keeping calibration production and image processing from being correctly separated.

            I'm working on what I'm tentatively calling a "gen3 crosstalk" for DECam, and my plan is to move to a Defects-like class.  I think this simplifies the on-disk representation by pushing common values/layers into metadata entries.  The new class should still be able to read the currently generated versions though, so the cp_pipe measure side can be done independently.

            Show
            czw Christopher Waters added a comment - I am 100% onboard with #1.  I think that's the only thing keeping calibration production and image processing from being correctly separated. I'm working on what I'm tentatively calling a "gen3 crosstalk" for DECam, and my plan is to move to a Defects-like class.  I think this simplifies the on-disk representation by pushing common values/layers into metadata entries.  The new class should still be able to read the currently generated versions though, so the cp_pipe measure side can be done independently.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            Meredith Rawls / DECam folk, I/we will obviously translate any DECam crosstalk objects to the new format, or make sure the old ones still work, of course.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - Meredith Rawls / DECam folk, I/we will obviously translate any DECam crosstalk objects to the new format, or make sure the old ones still work, of course.
            Hide
            mrawls Meredith Rawls added a comment -

            I am familiar with CrosstalkTask, subtractCrosstalk, getCrosstalk, and even prepCrosstalk, but not measureCrosstalk. I gather from the discussion that it is being used to generate crosstalk coefficients from some kind of calibration data, and the question is (1) where to put that operation and (2) how to store and use those coefficients?

            Assuming that is roughly correct, having (1) be cp_pipe seems like the right place. I support Christopher Waters's plan for (2) to have a Defects-like "curated calibration" class for storing crosstalk data, provided it can handle inter-CCD crosstalk (i.e., be DECam-friendly) and interface nicely with ip_isr.

            Show
            mrawls Meredith Rawls added a comment - I am familiar with CrosstalkTask, subtractCrosstalk, getCrosstalk, and even prepCrosstalk, but not measureCrosstalk. I gather from the discussion that it is being used to generate crosstalk coefficients from some kind of calibration data, and the question is (1) where to put that operation and (2) how to store and use those coefficients? Assuming that is roughly correct, having (1) be cp_pipe seems like the right place. I support Christopher Waters 's plan for (2) to have a Defects-like "curated calibration" class for storing crosstalk data, provided it can handle inter-CCD crosstalk (i.e., be DECam-friendly) and interface nicely with ip_isr.
            Hide
            mfisherlevine Merlin Fisher-Levine added a comment -

            It turns out that because Christopher Waters is overhauling all calibrations, creating a base class from which all calibrations products will inherit (which is AWESOME) this RFC is now superseded by a much better and more general plan, so is being withdrawn.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - It turns out that because Christopher Waters is overhauling all calibrations, creating a base class from which all calibrations products will inherit (which is AWESOME) this RFC is now superseded by a much better and more general plan, so is being withdrawn.

              People

              • Assignee:
                mfisherlevine Merlin Fisher-Levine
                Reporter:
                mfisherlevine Merlin Fisher-Levine
                Watchers:
                Andrés Alejandro Plazas Malagón, Christopher Waters, John Swinbank, Leanne Guy, Meredith Rawls, Merlin Fisher-Levine, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel