Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38948

Fix dataset query constraint bugs introduced on DM-38780

    XMLWordPrintable

    Details

      Description

      While starting DM-38498, I noticed that some logic involving the "dataset query constraint" option to QG generation should have been updated on DM-38780, but hadn't been (which is largely my fault, as the reviewer of that ticket just earlier the same day).  I had originally planned to put a fix on DM-38498 itself, but the original goal of that ticket now looks harder than it once did so I'm putting the fix here.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-38780 Modify graph builder so that it no longer uses unresolved refs

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Possible fix is on the branch.  Given the difficulty of adding unit tests on short notice (needs moderately complex data and tasks), I'm going to run some one-off tests this weekend against the USDF repos before putting it out for review.  I also want to fix the test coverage but don't want to get involved in that while the bug is on main.

             

            Show
            jbosch Jim Bosch added a comment - Possible fix is on the branch.  Given the difficulty of adding unit tests on short notice (needs moderately complex data and tasks), I'm going to run some one-off tests this weekend against the USDF repos before putting it out for review.  I also want to fix the test coverage but don't want to get involved in that while the bug is on main.  
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            One-off test was successful; the command I used to reproduce the problem is:

            $ pipetask qgraph -b /repo/main -i HSC/runs/RC2/w_2023_11/DM-38360 -p /sdf/group/rubin/sw/conda/envs/lsst-scipipe-6.0.0/share/eups/Linux64/drp_pipe/g10915be422+84f8699254/pipelines/HSC/DRP-RC2.yaml#step2cde -o u/jbosch/DM-38948

            Before this ticket this results in an exception.  After, I get:

            lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 869 quanta for 7 tasks, graph ID: '1682731670.6923134-3810612'
            		 
            Quanta           Tasks           
            ------ --------------------------
            		 
               432   finalizeCharacterization
            		 
                 1 fgcmBuildFromIsolatedStars
            		 
                 1               fgcmFitCycle
            		 
                 1         fgcmOutputProducts
            		 
               432         updateVisitSummary
            		 
                 1             makeVisitTable
            		 
                 1          makeCcdVisitTable

            which is the same as I get with w_2023_17, before the bug was introduced.

            Show
            jbosch Jim Bosch added a comment - One-off test was successful; the command I used to reproduce the problem is: $ pipetask qgraph -b /repo/main -i HSC/runs/RC2/w_2023_11/DM- 38360 -p /sdf/group/rubin/sw/conda/envs/lsst-scipipe- 6.0 . 0 /share/eups/Linux64/drp_pipe/g10915be422+84f8699254/pipelines/HSC/DRP-RC2.yaml#step2cde -o u/jbosch/DM- 38948 Before this ticket this results in an exception.  After, I get: lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 869 quanta for 7 tasks, graph ID: '1682731670.6923134-3810612' Quanta Tasks ------ -------------------------- 432 finalizeCharacterization 1 fgcmBuildFromIsolatedStars 1 fgcmFitCycle 1 fgcmOutputProducts 432 updateVisitSummary 1 makeVisitTable 1 makeCcdVisitTable which is the same as I get with w_2023_17, before the bug was introduced.
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            One-off tests were a bit easier than I thought and Jenkins is green, so it's time to put this in review.

            Only PR is https://github.com/lsst/pipe_base/pull/324

            I have some thoughts about how to improve test coverage that I'll put on another ticket.

            Show
            jbosch Jim Bosch added a comment - One-off tests were a bit easier than I thought and Jenkins is green, so it's time to put this in review. Only PR is https://github.com/lsst/pipe_base/pull/324 I have some thoughts about how to improve test coverage that I'll put on another ticket.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Looks good, thanks for fixing my mess.

            Show
            salnikov Andy Salnikov added a comment - Looks good, thanks for fixing my mess.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Jim Bosch
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.