Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19808

Avoid duplication in deferred prelight follow-up queries

    Details

      Description

      Building QuantumGraphs at even moderate scale with deferred dataset_id queries (as is required by SQLite) is quite slow, probably because we're running many more of these queries than we need.  See if we can cache the results in Python while iterating over results from the big query instead.

        Attachments

          Activity

          Hide
          jbosch Jim Bosch added a comment -

          It turns out this affects more than just deferred dataset ID queries - even when no dataset ID queries are deferred, we're still hitting the dataset table way more often than we need to in order to transform a dataset_id into a DatasetRef.

          Show
          jbosch Jim Bosch added a comment - It turns out this affects more than just deferred dataset ID queries - even when no dataset ID queries are deferred, we're still hitting the dataset table way more often than we need to in order to transform a dataset_id into a DatasetRef.
          Hide
          jbosch Jim Bosch added a comment -

          Andy Salnikov, I'm still doing a bit more profiling on this, but I think it's very unlikely that the changes I've already made will be backed out as a result of that, and hence it should be safe to start the review now.  All changes in daf_butler, at https://github.com/lsst/daf_butler/pull/163.

          Show
          jbosch Jim Bosch added a comment - Andy Salnikov , I'm still doing a bit more profiling on this, but I think it's very unlikely that the changes I've already made will be backed out as a result of that, and hence it should be safe to start the review now.  All changes in daf_butler, at https://github.com/lsst/daf_butler/pull/163 .
          Hide
          salnikov Andy Salnikov added a comment -

          Looks OK, few comments on PR.

          Show
          salnikov Andy Salnikov added a comment - Looks OK, few comments on PR.

            People

            • Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Jim Bosch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel