Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35883

CrosstalkCalib datasets may have ambiguous amplifier ordering for coefficients.

    XMLWordPrintable

Details

    • Improvement
    • Status: Invalid
    • Resolution: Done
    • None
    • cp_pipe

    Description

      CrosstalkCalib datasets generated by CrosstalkSolveTask may have ambiguous ordering of the amplifiers, resulting in crosstalk matrices that do not have the correct mapping of index to segment.  This is caused by the use of

      ordering = list(ratios.keys())
      

      in line 575 of measureCrosstalk.py to convert the ratio keys (target segment names) to a list.  This conversion is not guaranteed to preserve the dictionary key ordering and depends on the order of key-pair insertion and on the latest modifications to the dictionary.

      Attachments

        Issue Links

          Activity

            snyder18 Adam Snyder added a comment -

            If the standard dictionary is order, then is there a change that the file read/write is causing the dictionary keys to be reordered?  

            snyder18 Adam Snyder added a comment - If the standard dictionary is order, then is there a change that the file read/write is causing the dictionary keys to be reordered?  
            tjenness Tim Jenness added a comment -

            Aha. There is a semi-hidden parameter for yaml.dump:

            yaml.dump(data, sort_keys=False)
            

            So it seems by default yaml always writes content by sorting the dicts... Relying on sort order round-tripping through YAML seems like a bad idea. You may need to reconsider the data structure you are serializing so that it stores the relationships somewhere without relying on dict ordering.

            tjenness Tim Jenness added a comment - Aha. There is a semi-hidden parameter for yaml.dump: yaml.dump(data, sort_keys = False ) So it seems by default yaml always writes content by sorting the dicts... Relying on sort order round-tripping through YAML seems like a bad idea. You may need to reconsider the data structure you are serializing so that it stores the relationships somewhere without relying on dict ordering.
            snyder18 Adam Snyder added a comment -

            Hmm, I just opened another issue related to the ratio data structure itself, in order to mitigate long file read times, so perhaps this issue would be mitigated by those changes, which I propose be to just JSON as the file type for the CrosstalkRatios.

            snyder18 Adam Snyder added a comment - Hmm, I just opened another issue related to the ratio data structure itself, in order to mitigate long file read times, so perhaps this issue would be mitigated by those changes, which I propose be to just JSON as the file type for the CrosstalkRatios.
            tjenness Tim Jenness added a comment -

            Yes, I saw that other ticket. I do think you need to design a data structure that does not rely on key ordering.

            tjenness Tim Jenness added a comment - Yes, I saw that other ticket. I do think you need to design a data structure that does not rely on key ordering.

            Marking this ticket as invalid, as the fix for this problem will be part of DM-37819.

            czw Christopher Waters added a comment - Marking this ticket as invalid, as the fix for this problem will be part of DM-37819 .

            People

              Unassigned Unassigned
              snyder18 Adam Snyder
              Adam Snyder, Christopher Waters, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.