Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: jointcal
-
Labels:None
-
Story Points:1
-
Sprint:AP S19-5, AP S19-6
-
Team:External
Description
Paul Price noticed that the use of unsigned (== uint32, usually) as the index type for Eigen matrices in jointcal might be a problem:
- Eigen docs appear to want a signed integer;
- our number of sources in some of our processing is approaching 2^31.
There's no hard evidence that this is what's causing the segfault we're seeing when processing the (larger, still-proprietary) HSC UltraDeep, and since unsigned should be able to handle 2^32, I'm personally skeptical that this is the problem. But it should be easy to fix, and a good thing to do regardless, especially for the first reason.
std::ptrdiff_t is probably the right choice to use instead - that will be a signed integer as large as the pointer type (i.e. int64, usually), and hence the largest unsigned type that's actually usable as an index.
Attachments
Issue Links
- blocks
-
DM-17743 Shepherd tickets for HSC release candidate
- To Do
Activity
Field | Original Value | New Value |
---|---|---|
Status | To Do [ 10001 ] | In Progress [ 3 ] |
Watchers | Jim Bosch, John Swinbank [ Jim Bosch, John Swinbank ] | Jim Bosch, John Swinbank, Paul Price, Yusra AlSayyad [ Jim Bosch, John Swinbank, Paul Price, Yusra AlSayyad ] |
Reviewers | Paul Price [ price ] | |
Status | In Progress [ 3 ] | In Review [ 10004 ] |
Sprint | AP S19-5 [ 833 ] | |
Priority | Undefined [ 10000 ] | Major [ 3 ] |
Attachment | cholmod-longs.patch [ 37848 ] |
Reviewers | Paul Price [ price ] | John Parejko [ parejkoj ] |
Assignee | John Parejko [ parejkoj ] | Paul Price [ price ] |
Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
Sprint | AP S19-5 [ 833 ] | AP S19-5, AP S19-6 [ 833, 834 ] |
Team | Alert Production [ 10300 ] | External [ 12117 ] |
Resolution | Done [ 10000 ] | |
Status | Reviewed [ 10101 ] | Done [ 10002 ] |
Paul Price: I've gotten this to build locally without warnings (Ubuntu 18.04), and I think I caught all of the necessary places where it might matter what the matrix size was. Can you please both review the PR, and see if it resolves the segfault?