Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23992

Cannot load refcats in Gen 3 DECam processing

    XMLWordPrintable

Details

    • 2
    • DRP S20-5 (Apr)
    • Data Release Production
    • No

    Description

      Attempting to run the Gen 3 single-frame pipeline as described in DM-23616 leads to AstrometryTask crashing with the error "No reference tables could be found for input region". Running with --show graph reports that no photometric or astrometric reference catalog shards are included in the quanta for CalibrateTask. However, they can be retrieved from the Butler using Registry.queryDatasets (only tested with no data ID constraints).

      The working hypothesis from #dm-middleware is that this is a problem in ingestion (i.e., in convert_gen2_repo_to_gen3.py), so I'm filing this as a bug in obs_base.

      Please fix the conversion(?) so that a repository created as described in DM-23616 can correctly match catalogs to image data IDs.

      Attachments

        Issue Links

          Activity

            krzys Krzysztof Findeisen added a comment - - edited

            Because of changes to the config, the workaround for the original bug is now config.defineVisits.computeVisitRegions.active.padding = 32768.

            krzys Krzysztof Findeisen added a comment - - edited Because of changes to the config, the workaround for the original bug is now config.defineVisits.computeVisitRegions.active.padding = 32768 .
            nlust Nate Lust added a comment -

            I merged the small commits to unblock AP. Any future work around this topic should happen on a new ticket.

            nlust Nate Lust added a comment - I merged the small commits to unblock AP. Any future work around this topic should happen on a new ticket.

            Hey folks — much appreciated if you can add story point estimates to tickets, please. Thanks!

            swinbank John Swinbank added a comment - Hey folks — much appreciated if you can add story point estimates to tickets, please. Thanks!

            The issue that nlust saw with trying to load reference objects by physical filter is still an issue. Is the preferred solution still to add all physical -> band combinations to the filter map (I'll make a new ticket if so), or has any other solution evolved in the meantime? I don't understand how any gen3 decam runs are working without that fix.

            ctslater Colin Slater added a comment - The issue that nlust saw with trying to load reference objects by physical filter is still an issue. Is the preferred solution still to add all physical -> band combinations to the filter map (I'll make a new ticket if so), or has any other solution evolved in the meantime? I don't understand how any gen3 decam runs are working without that fix.
            Parejkoj John Parejko added a comment - - edited

            Which refcat is the problem here? If it's PS1, then yes, I think the correct thing is to add the "DECam physical"->"PS1 band" to the decam filterMap. Although I'd be happy if someone corrected me to show that the code now uses the FilterLabel.band() when determining what refcat filter to load.

            If it's Gaia, then we want DM-27843 so we only have to specify one config (which is gen2/3 independent).

            Parejkoj John Parejko added a comment - - edited Which refcat is the problem here? If it's PS1, then yes, I think the correct thing is to add the "DECam physical"->"PS1 band" to the decam filterMap. Although I'd be happy if someone corrected me to show that the code now uses the FilterLabel.band() when determining what refcat filter to load. If it's Gaia, then we want DM-27843 so we only have to specify one config (which is gen2/3 independent).

            People

              nlust Nate Lust
              krzys Krzysztof Findeisen
              Colin Slater, Dino Bektesevic, Jim Bosch, John Parejko, Krzysztof Findeisen, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.