Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18895

Use std::ptrdiff_t as index type in jointcal Eigen objects

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jointcal
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      AP S19-5, AP S19-6
    • Team:
      External

      Description

      Paul Price noticed that the use of unsigned (== uint32, usually) as the index type for Eigen matrices in jointcal might be a problem:

      • Eigen docs appear to want a signed integer;
      • our number of sources in some of our processing is approaching 2^31.

      There's no hard evidence that this is what's causing the segfault we're seeing when processing the (larger, still-proprietary) HSC UltraDeep, and since unsigned should be able to handle 2^32, I'm personally skeptical that this is the problem.  But it should be easy to fix, and a good thing to do regardless, especially for the first reason.

      std::ptrdiff_t is probably the right choice to use instead - that will be a signed integer as large as the pointer type (i.e. int64, usually), and hence the largest unsigned type that's actually usable as an index.

       

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Status To Do [ 10001 ] In Progress [ 3 ]
            Parejkoj John Parejko made changes -
            Watchers Jim Bosch, John Swinbank [ Jim Bosch, John Swinbank ] Jim Bosch, John Swinbank, Paul Price, Yusra AlSayyad [ Jim Bosch, John Swinbank, Paul Price, Yusra AlSayyad ]
            Parejkoj John Parejko made changes -
            Reviewers Paul Price [ price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Parejkoj John Parejko made changes -
            Sprint AP S19-5 [ 833 ]
            Priority Undefined [ 10000 ] Major [ 3 ]
            yusra Yusra AlSayyad made changes -
            Link This issue blocks DM-17743 [ DM-17743 ]
            jbosch Jim Bosch made changes -
            Attachment cholmod-longs.patch [ 37848 ]
            Parejkoj John Parejko made changes -
            Reviewers Paul Price [ price ] John Parejko [ parejkoj ]
            Parejkoj John Parejko made changes -
            Assignee John Parejko [ parejkoj ] Paul Price [ price ]
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            swinbank John Swinbank made changes -
            Sprint AP S19-5 [ 833 ] AP S19-5, AP S19-6 [ 833, 834 ]
            swinbank John Swinbank made changes -
            Team Alert Production [ 10300 ] External [ 12117 ]
            price Paul Price made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                price Paul Price
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                John Parejko
                Watchers:
                Jim Bosch, John Parejko, John Swinbank, Paul Price, Tim Jenness, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel