Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22708

Fix decam gen3 ingest

    XMLWordPrintable

    Details

      Description

      It turns out the work I did on decam for DM-20763 was not complete: I only tested that one dataId could be retrieved from the ingested raw data, but there are two ccds in that file in two different HDUs and the second one is not actually getting ingested. The gen2 CameraMapper path for a decam raw looks like decam%(visit)07d.fits.fz[%(hdu)d], so the hdu number is baked in there. As far as I can tell, gen3 doesn't have any way to encode hdu number. Maybe this has to be a further specialization in the RawFormatter. File paths that go into butler.ingest have to actually exist, so we will have to do something to the raw paths that we pass into `butler.ingest`.

      The gen2 butler registry has an `hdu` field that encodes which hdu to get that raw from. It doesn't look like there's an equivalent `hdu` field anywhere in the gen3 registry: could we add an extra `hdu` field to the `posix_datastore_records` table? That's my best guess as to where it should live.

        Attachments

          Issue Links

            Activity

            Parejkoj John Parejko created issue -
            Parejkoj John Parejko made changes -
            Field Original Value New Value
            Link This issue is triggered by DM-20763 [ DM-20763 ]
            Parejkoj John Parejko made changes -
            Component/s daf_butler [ 11300 ]
            Component/s Middleware [ 10707 ]
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-21862 [ DM-21862 ]
            Parejkoj John Parejko made changes -
            Link This issue blocks DM-22655 [ DM-22655 ]
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            swinbank John Swinbank made changes -
            Epic Link DM-22633 [ 427742 ]
            swinbank John Swinbank made changes -
            Sprint AP S20-1 (December) [ 981 ]
            swinbank John Swinbank made changes -
            Sprint AP S20-2 (January) [ 983 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-23024 [ DM-23024 ]
            Parejkoj John Parejko made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Parejkoj John Parejko made changes -
            Attachment gen3ingest.dot.pdf [ 42231 ]
            Parejkoj John Parejko made changes -
            Attachment gen3ingest.dot [ 42236 ]
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            Parejkoj John Parejko made changes -
            Link This issue is triggering DM-23249 [ DM-23249 ]
            swinbank John Swinbank made changes -
            Story Points 8
            Urgent? off

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Tim Jenness
              Watchers:
              Colin Slater, Jim Bosch, John Parejko, John Swinbank, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.