Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-29256

Error reading in Quantum Graph with measurement config

    XMLWordPrintable

    Details

    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      When submitting the attached BPS config file, an Exception is thrown when trying to read the generated quantum graph. The quantum graph is generated with a config file that configures a measurement plugin that is not in `lsst_distrib`. The creation and running of the Quantum Graph works from the command line but fails when trying to submit jobs via BPS.

        Attachments

          Issue Links

            Activity

            Hide
            nlust Nate Lust added a comment - - edited

            Arun Kannawadi can you point me to where the test quantum graph is on disk?
            [edit] - nevermind I overlooked a comment I see it now.

            Show
            nlust Nate Lust added a comment - - edited Arun Kannawadi can you point me to where the test quantum graph is on disk? [edit] - nevermind I overlooked a comment I see it now.
            Hide
            kannawad Arun Kannawadi added a comment -

            The path earlier (fixed now) was incorrect and you can fin the quantum graph at /project/kannawad/gaapCLI.qgraph. It can also be generated using Michelle's comment.

            Show
            kannawad Arun Kannawadi added a comment - The path earlier (fixed now) was incorrect and you can fin the quantum graph at /project/kannawad/gaapCLI.qgraph. It can also be generated using Michelle's comment.
            Hide
            nlust Nate Lust added a comment -

            So here is the issue, and it is not entirely unexpected. The quantum graph contains a pickled version of the configs. With packages in the stack these are already setup and unpickling can pull all the appropriate bits in. However, external packages that register something have a side effect on the config when they are created, but if they are not imported in the new environment, do not get a change to fire their "import time" behavior prior to the config being persisted.

            Normally this is mitigated by having imports at the top of persisted config files, but in the case of the quantum graph, the object has only pickles that don't go through the normal load process.

            I have a few ideas to mitigate this issue when I get a few moments, I will look at implementing them.

            Show
            nlust Nate Lust added a comment - So here is the issue, and it is not entirely unexpected. The quantum graph contains a pickled version of the configs. With packages in the stack these are already setup and unpickling can pull all the appropriate bits in. However, external packages that register something have a side effect on the config when they are created, but if they are not imported in the new environment, do not get a change to fire their "import time" behavior prior to the config being persisted. Normally this is mitigated by having imports at the top of persisted config files, but in the case of the quantum graph, the object has only pickles that don't go through the normal load process. I have a few ideas to mitigate this issue when I get a few moments, I will look at implementing them.
            Hide
            mgower Michelle Gower added a comment -

            I was browsing tickets with component ctrl_bps and ran across this one.  Is this still a problem?

            Show
            mgower Michelle Gower added a comment - I was browsing tickets with component ctrl_bps and ran across this one.  Is this still a problem?
            Hide
            kannawad Arun Kannawadi added a comment -

            It's not a problem any more in the sense that `meas_extensions_gaap` is now part of `lsst_distrib`. However, I don't know if there has been changes to ctrl_bps that will avoid this problem for any new plugin that may be developed. I could check with a dummy plugin if we want to mark it as Invalid or not.

            Show
            kannawad Arun Kannawadi added a comment - It's not a problem any more in the sense that `meas_extensions_gaap` is now part of `lsst_distrib`. However, I don't know if there has been changes to ctrl_bps that will avoid this problem for any new plugin that may be developed. I could check with a dummy plugin if we want to mark it as Invalid or not.

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              kannawad Arun Kannawadi
              Watchers:
              Arun Kannawadi, Michelle Gower, Nate Lust
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.