Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26181

Ensure that filters are defined in pipetask multiprocessing

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Sprint:
      DB_F20_06
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      In DM-26119 I learned that when pipetask uses multiprocessing we never instantiate an Instrument and therefore we never define the filters for that instrument in the global singleton.

      In single process mode it's fine because at some point an Instrument is created.

      Modify pipetask multiprocessing such that the dataIds are scanned for the "instrument" dimension and we call Instrument.fromName(dataId["instrument"], registry). Currently we only expect one instrument.

      When the singleton is removed it's likely that some related initialization will be needed to register the filters but we assume that would allow the same initialization for multiple instruments.

      CC/ Krzysztof Findeisen, John Parejko in case they have come across this problem before.

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26119 Test dataset disassembly with ci_hsc_gen3

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-26069 Create requirements list for new Filter implementation

          • Done

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-624 Replace afw.image.Filter with global-free label-only classes

          • Retired

            Activity

            Ascending order - Click to sort in descending order
            6 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Yes, that is probably expected. I don't think there is anything we can do about it at the moment. I think what you have done will make a huge improvement to consistency of execution.

            Show
            tjenness Tim Jenness added a comment - Yes, that is probably expected. I don't think there is anything we can do about it at the moment. I think what you have done will make a huge improvement to consistency of execution.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            OK, thanks, I think it is ready for review then. Jenkins has just started, I'll wait until it finishes.

            Eli Rykoff This should fix both single- and multi-process cases.

            Show
            salnikov Andy Salnikov added a comment - OK, thanks, I think it is ready for review then. Jenkins has just started, I'll wait until it finishes. Eli Rykoff This should fix both single- and multi-process cases.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            JIRA is slow as usual, PR is here: https://github.com/lsst/ctrl_mpexec/pull/62

            Show
            salnikov Andy Salnikov added a comment - JIRA is slow as usual, PR is here: https://github.com/lsst/ctrl_mpexec/pull/62
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Looks great.

            Show
            tjenness Tim Jenness added a comment - Looks great.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Thanks, I added assert message, and I also had to add a (ugly) monkey-patch to unit test, I missed unit test failure in previous commit somehow (but Jenkins alerted me). You can check PR again to see if it is OK with you.

            Show
            salnikov Andy Salnikov added a comment - Thanks, I added assert message, and I also had to add a (ugly) monkey-patch to unit test, I missed unit test failure in previous commit somehow (but Jenkins alerted me). You can check PR again to see if it is OK with you.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Eli Rykoff, John Parejko, Krzysztof Findeisen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.