Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19988

fewer reference catalogs for a ccd if a tract is specified in making quantum graph

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      The ref cat quanta in the generated quantum graph can be different for processing a CCD in CalibrateTask, depending on whether the data selection query includes a tract. For example,

      pipetask -d "visit=1294 and detector=34"
      

      versus

      pipetask -d "visit=1294 and detector=34 and tract=9615"
      

      with the HSC-RC2 repo, the predictedInputs of CalibrateTask has two ref_cat skypix: 222341 and skypix: 222348 in the former, but has only one ref_cat skypix: 222341 in the latter.

      In the latter, CalibrateTask got fewer reference objects and then fails with "RuntimeError: Unable to match sources" in astrometry matchObjectsToSources. w_2019_22 was used in my testing.

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            Reviewed/approved all changes, code looks super clean, almost nothing to comment on I think I understand how it works reading comments and docstrings. I'm not super happy that pipe_base is exposed to sqlalchemy stuff now, but it's probably not the last iteration of it, we can still improve things.

            Show
            salnikov Andy Salnikov added a comment - Reviewed/approved all changes, code looks super clean, almost nothing to comment on I think I understand how it works reading comments and docstrings. I'm not super happy that pipe_base is exposed to sqlalchemy stuff now, but it's probably not the last iteration of it, we can still improve things.
            Hide
            jbosch Jim Bosch added a comment -

            I've just pushed a few more changes.  In addition to addressing review comments (all pretty straightforward),

            • The daf_butler changes broke a ci_hsc unit test, so there's now a ci_hsc branch to rewrite that test function.  Happily this also provides a bit more test coverage of the changes in daf_butler.
            • Rewriting that test (and thinking about your comment about pipe_base being exposed to sqlalchemy) inspired me to add a whereDataId method to QueryBuilder to keep the sqlalchemy logic contained in daf_butler.
            • Running that test uncovered a long-standing problem in how SqlRegistryDatabaseDict uses transactions.

            So, upshot is that there are now three new commits on daf_butler (one a trivial doc fix), one more in pipe_base, and one in ci_hscAndy Salnikov, could you take a quick look at those?

            Show
            jbosch Jim Bosch added a comment - I've just pushed a few more changes.  In addition to addressing review comments (all pretty straightforward), The daf_butler changes broke a ci_hsc unit test, so there's now a ci_hsc branch to rewrite that test function.  Happily this also provides a bit more test coverage of the changes in daf_butler. Rewriting that test (and thinking about your comment about pipe_base being exposed to sqlalchemy) inspired me to add a whereDataId method to QueryBuilder to keep the sqlalchemy logic contained in daf_butler. Running that test uncovered a long-standing problem in how  SqlRegistryDatabaseDict uses transactions. So, upshot is that there are now three new commits on daf_butler (one a trivial doc fix), one more in pipe_base, and one in ci_hsc .  Andy Salnikov , could you take a quick look at those?
            Hide
            salnikov Andy Salnikov added a comment -

            I checked new commits, looks good to me.

            Show
            salnikov Andy Salnikov added a comment - I checked new commits, looks good to me.
            Hide
            jbosch Jim Bosch added a comment -

            Merged to master.

            This includes a branch for ci_hsc_gen2, which is not yet being advertised for general use but would have been broken by this change (the fix is just a cherry-pick of the commit for ci_hsc).

             

            Show
            jbosch Jim Bosch added a comment - Merged to master. This includes a branch for ci_hsc_gen2, which is not yet being advertised for general use but would have been broken by this change (the fix is just a cherry-pick of the commit for ci_hsc).  
            Hide
            swinbank John Swinbank added a comment -

            Hi folks — a reminder to please make sure completed tickets have appropriate epics and SP counts attached. Thanks!

            Show
            swinbank John Swinbank added a comment - Hi folks — a reminder to please make sure completed tickets have appropriate epics and SP counts attached. Thanks!

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Christopher Stephens [X] (Inactive), Hsin-Fang Chiang, Jim Bosch, John Swinbank, Jonathan Sick, Michelle Gower
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.