Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18895

Use std::ptrdiff_t as index type in jointcal Eigen objects

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jointcal
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP S19-5, AP S19-6
    • Team:
      External

      Description

      Paul Price noticed that the use of unsigned (== uint32, usually) as the index type for Eigen matrices in jointcal might be a problem:

      • Eigen docs appear to want a signed integer;
      • our number of sources in some of our processing is approaching 2^31.

      There's no hard evidence that this is what's causing the segfault we're seeing when processing the (larger, still-proprietary) HSC UltraDeep, and since unsigned should be able to handle 2^32, I'm personally skeptical that this is the problem.  But it should be easy to fix, and a good thing to do regardless, especially for the first reason.

      std::ptrdiff_t is probably the right choice to use instead - that will be a signed integer as large as the pointer type (i.e. int64, usually), and hence the largest unsigned type that's actually usable as an index.

       

        Attachments

          Issue Links

            Activity

            No work has yet been logged on this issue.

              People

              • Assignee:
                price Paul Price
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                John Parejko
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Paul Price, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel