# Move crosstalk measurement code and change data format.

XMLWordPrintable

## Details

• Type: RFC
• Status: Withdrawn
• Resolution: Done
• Component/s:
• Labels:
None

## Description

1) measureCrosstalk.py should live in cp_pipe now that it exists, as it's creating a  calibration product, so step 1 is to simply move it there, and move the writeCrosstalkCoeffs method as well, as that really belongs with the measuring code.

2) Update the data format for the crosstalk coefficients. Current format written is a 4-deep nested dict, with the two outer layers being 100% superfluous. The proposed change to the format will either be to just rip these out, or better, make a python defects class holding the coefficients so that they can be used by ampName (currently they're stored like that, but during the loading the names are discarded and they're thrown into a numpy array, losing the safety that the dict keys provide), which would involve making the requisite changes to the isrTask to enable their use.

## Activity

Hide
Tim Jenness added a comment -

Are these implemented in a CrossTalk class that has read/write methods for serialization? Or are they used as python dicts directly?

Show
Tim Jenness added a comment - Are these implemented in a CrossTalk class that has read/write methods for serialization? Or are they used as python dicts directly?
Hide
Merlin Fisher-Levine added a comment -

At the moment, neither. There's no class, and a dict of dicts of dicts of dicts is written manually in yaml, and when retrieved, the two outer layers are discarded, and the inner two, which give the coefficients by name, have their naming-protection discarded, and the values shoved into a naked numpy array.

Show
Merlin Fisher-Levine added a comment - At the moment, neither. There's no class, and a dict of dicts of dicts of dicts is written manually in yaml, and when retrieved, the two outer layers are discarded, and the inner two, which give the coefficients by name, have their naming-protection discarded, and the values shoved into a naked numpy array.
Hide
Andrés Alejandro Plazas Malagón added a comment -

Example:

 /project/shared/auxTel/CALIB/crosstalk/crosstalk-S00-det000.yaml 

 {'crosstalk': {'auxTel': {'C10': {'C10': 0.0,  'C11': 0.00041452,  'C12': 5.3125e-05,  'C13': 1.0496e-05,  'C14': -7.5229e-06,  'C15': -2.0852e-05,  'C16': -3.0844e-05,  'C17': -2.7888e-05,  'C07': -3.1578e-05,  'C06': -3.738e-05,  'C05': -3.806e-05,  'C04': -4.1003e-05,  'C03': -3.189e-05,  'C02': -3.4353e-05,  'C01': -3.5075e-05,  'C00': -3.8668e-05},  'C11': {'C10': 0.00029295,  'C11': 0.0,  'C12': 0.00039701,  'C13': 0.0001004, 

Show
Andrés Alejandro Plazas Malagón added a comment - Example: /project/shared/auxTel/CALIB/crosstalk/crosstalk-S00-det000.yaml {'crosstalk': {'auxTel': {'C10': {'C10': 0.0, 'C11': 0.00041452, 'C12': 5.3125e-05, 'C13': 1.0496e-05, 'C14': -7.5229e-06, 'C15': -2.0852e-05, 'C16': -3.0844e-05, 'C17': -2.7888e-05, 'C07': -3.1578e-05, 'C06': -3.738e-05, 'C05': -3.806e-05, 'C04': -4.1003e-05, 'C03': -3.189e-05, 'C02': -3.4353e-05, 'C01': -3.5075e-05, 'C00': -3.8668e-05}, 'C11': {'C10': 0.00029295, 'C11': 0.0, 'C12': 0.00039701, 'C13': 0.0001004,
Hide
Christopher Waters added a comment -

I am 100% onboard with #1.  I think that's the only thing keeping calibration production and image processing from being correctly separated.

I'm working on what I'm tentatively calling a "gen3 crosstalk" for DECam, and my plan is to move to a Defects-like class.  I think this simplifies the on-disk representation by pushing common values/layers into metadata entries.  The new class should still be able to read the currently generated versions though, so the cp_pipe measure side can be done independently.

Show
Christopher Waters added a comment - I am 100% onboard with #1.  I think that's the only thing keeping calibration production and image processing from being correctly separated. I'm working on what I'm tentatively calling a "gen3 crosstalk" for DECam, and my plan is to move to a Defects-like class.  I think this simplifies the on-disk representation by pushing common values/layers into metadata entries.  The new class should still be able to read the currently generated versions though, so the cp_pipe measure side can be done independently.
Hide
Merlin Fisher-Levine added a comment -

Meredith Rawls / DECam folk, I/we will obviously translate any DECam crosstalk objects to the new format, or make sure the old ones still work, of course.

Show
Merlin Fisher-Levine added a comment - Meredith Rawls / DECam folk, I/we will obviously translate any DECam crosstalk objects to the new format, or make sure the old ones still work, of course.
Hide
Meredith Rawls added a comment -

I am familiar with CrosstalkTask, subtractCrosstalk, getCrosstalk, and even prepCrosstalk, but not measureCrosstalk. I gather from the discussion that it is being used to generate crosstalk coefficients from some kind of calibration data, and the question is (1) where to put that operation and (2) how to store and use those coefficients?

Assuming that is roughly correct, having (1) be cp_pipe seems like the right place. I support Christopher Waters's plan for (2) to have a Defects-like "curated calibration" class for storing crosstalk data, provided it can handle inter-CCD crosstalk (i.e., be DECam-friendly) and interface nicely with ip_isr.

Show
Meredith Rawls added a comment - I am familiar with CrosstalkTask, subtractCrosstalk, getCrosstalk, and even prepCrosstalk, but not measureCrosstalk. I gather from the discussion that it is being used to generate crosstalk coefficients from some kind of calibration data, and the question is (1) where to put that operation and (2) how to store and use those coefficients? Assuming that is roughly correct, having (1) be cp_pipe seems like the right place. I support Christopher Waters 's plan for (2) to have a Defects-like "curated calibration" class for storing crosstalk data, provided it can handle inter-CCD crosstalk (i.e., be DECam-friendly) and interface nicely with ip_isr.
Hide
Merlin Fisher-Levine added a comment -

It turns out that because Christopher Waters is overhauling all calibrations, creating a base class from which all calibrations products will inherit (which is AWESOME) this RFC is now superseded by a much better and more general plan, so is being withdrawn.

Show
Merlin Fisher-Levine added a comment - It turns out that because Christopher Waters is overhauling all calibrations, creating a base class from which all calibrations products will inherit (which is AWESOME) this RFC is now superseded by a much better and more general plan, so is being withdrawn.

## People

• Assignee:
Merlin Fisher-Levine
Reporter:
Merlin Fisher-Levine
Watchers:
Andrés Alejandro Plazas Malagón, Christopher Waters, John Swinbank, Leanne Guy, Meredith Rawls, Merlin Fisher-Levine, Tim Jenness