Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: jointcal
-
Labels:None
-
Story Points:1
-
Sprint:AP S19-5, AP S19-6
-
Team:External
Description
Paul Price noticed that the use of unsigned (== uint32, usually) as the index type for Eigen matrices in jointcal might be a problem:
- Eigen docs appear to want a signed integer;
- our number of sources in some of our processing is approaching 2^31.
There's no hard evidence that this is what's causing the segfault we're seeing when processing the (larger, still-proprietary) HSC UltraDeep, and since unsigned should be able to handle 2^32, I'm personally skeptical that this is the problem. But it should be easy to fix, and a good thing to do regardless, especially for the first reason.
std::ptrdiff_t is probably the right choice to use instead - that will be a signed integer as large as the pointer type (i.e. int64, usually), and hence the largest unsigned type that's actually usable as an index.
Attachments
Issue Links
- blocks
-
DM-17743 Shepherd tickets for HSC release candidate
- To Do
Oh, and please make the new PRs (from your moved branches to master) match the dev guide so the goal is more obvious ("DM-NNNN: JIRA Ticket Title").