Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22708

Fix decam gen3 ingest

    XMLWordPrintable

    Details

      Description

      It turns out the work I did on decam for DM-20763 was not complete: I only tested that one dataId could be retrieved from the ingested raw data, but there are two ccds in that file in two different HDUs and the second one is not actually getting ingested. The gen2 CameraMapper path for a decam raw looks like decam%(visit)07d.fits.fz[%(hdu)d], so the hdu number is baked in there. As far as I can tell, gen3 doesn't have any way to encode hdu number. Maybe this has to be a further specialization in the RawFormatter. File paths that go into butler.ingest have to actually exist, so we will have to do something to the raw paths that we pass into `butler.ingest`.

      The gen2 butler registry has an `hdu` field that encodes which hdu to get that raw from. It doesn't look like there's an equivalent `hdu` field anywhere in the gen3 registry: could we add an extra `hdu` field to the `posix_datastore_records` table? That's my best guess as to where it should live.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-21862 Extend Gen3 butler support for obs_decam

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-22655 Genericize gen2to3.py to be useable with any gen2 repo

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-23024 Support multi-dataset single file ingest in daf_butler

          • Done
          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20763 Add initial support for Gen3 Butler to obs_decam

          • Done
          is triggering

          Bug - A problem which impairs or prevents the functions of the product. DM-23249 New decam ingest tests need skipif for testdata_decam

          • Done

            Activity

            Ascending order - Click to sort in descending order
            8 older comments
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            That certainly is part of it. I don't know if it's all of it though. 6x slower is quite a lot.

            Show
            Parejkoj John Parejko added a comment - That certainly is part of it. I don't know if it's all of it though. 6x slower is quite a lot.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            3 times isn't it?

            Show
            tjenness Tim Jenness added a comment - 3 times isn't it?
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            gen3 is 180s for just raws. gen2 is 60s for raw+calib+defect, and by-eye timing the raws are about half of that.

            Show
            Parejkoj John Parejko added a comment - gen3 is 180s for just raws. gen2 is 60s for raw+calib+defect, and by-eye timing the raws are about half of that.
            Hide
            Permalink
            tjenness Tim Jenness added a comment - - edited

            In theory we could modify ObservationInfo constructor so that you could select which properties you want calculated (or allow all the translations to be on demand). That would at least allow ingest to only ask for the handful of items it needs. I don't think gen2 ingest was switched over to use ObservationInfo yet (obs_lsst does).

            Show
            tjenness Tim Jenness added a comment - - edited In theory we could modify ObservationInfo constructor so that you could select which properties you want calculated (or allow all the translations to be on demand). That would at least allow ingest to only ask for the handful of items it needs. I don't think gen2 ingest was switched over to use ObservationInfo yet (obs_lsst does).
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Thanks for the review comments. I believe I've addressed them all.

            New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31113/pipeline

            Show
            Parejkoj John Parejko added a comment - Thanks for the review comments. I believe I've addressed them all. New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/31113/pipeline

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Tim Jenness
              Watchers:
              Colin Slater, Jim Bosch, John Parejko, John Swinbank, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.