Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18695

Make afw.catalog.readFits threadsafe

    Details

    • Templates:

      Description

      As part of my attempts at speeding up reading data from all visits in a tract, I've releasing the gil in afw.catalog.readFits and use a ThreadPool to try reading multiple files at once (as opposed to a ProcessPool, which requires pickle/unpickle and our Catalog pickle code is slow and probably difficult to speed up). This causes segfaults. Krzysztof Findeisen and I spent a couple of hours yesterday digging into it, but didn't find an obvious source of static class members or global variables, beyond the daf.base.Citizen object, but none of the tracebacks I've seen reference Citizen that I can tell.

      I'll add an sample test script to this ticket, and push a branch, if anyone else wants to poke around. Debugging concurrency issues is "fun".

      I've tagged a bunch of the people with C++ expertise that I could think of: if you're uninterested, please remove yourself; if you think someone else might have expertise to add, please add them!

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  Parejkoj John Parejko
                  Watchers:
                  Christopher Waters, Colin Slater, Fritz Mueller, Gabor Kovacs, Jim Bosch, John Parejko, John Swinbank, Kian-Tat Lim, Krzysztof Findeisen, Paul Price, Yusra AlSayyad
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  11 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Summary Panel