Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23445

Move visit definition out of raw ingest for Gen3

    XMLWordPrintable

    Details

    • Story Points:
      0
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      At present, RawIngestTask is responsible for adding three different types of dimension records:

      • exposure
      • visit
      • visit_detector_region

      We should move the latter two to a separate script that is run after RawIngestTask and (at least at present) only after all associated raws for a visit have been ingested. That will open the door to (later) being able to define different visit definitions for the same exposure (DM-15536).

      This will not on its own allow individual raws to be ingested incrementally - RawIngestTask still needs all raws for an exposure to be present in a single invocation so we can add the exposure dimension record once without needing to see if it already exists in the Registry. That could be fixed by using Database.sync in a new high-level counterpart to Registry.insertDimensionData that does perform such a check, but we'll first need to figure out how to handle transactions in ingest - Database.sync is not and probably cannot be part of a larger transaction.

      But even that is an easier problem to solve than the corresponding one for visits, because (in addition to the synchronization problem) the visit dimension record needs to contain a region that is currently computed from the regions of all of the detectors that went into it (but could be computed from cameraGeom in the future).

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue blocks DM-15536 [ DM-15536 ]
            swinbank John Swinbank made changes -
            Team Data Access and Database [ 10204 ]
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-23428 [ DM-23428 ]
            mgower Michelle Gower made changes -
            Link This issue blocks DM-23843 [ DM-23843 ]
            jbosch Jim Bosch made changes -
            Assignee Jim Bosch [ jbosch ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-15536 [ DM-15536 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-15536 [ DM-15536 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-23208 [ DM-23208 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-23208 [ DM-23208 ]
            jbosch Jim Bosch made changes -
            Story Points 2 0
            jbosch Jim Bosch made changes -
            Reviewers Nate Lust, Tim Jenness [ nlust, tjenness ]
            jbosch Jim Bosch made changes -
            Component/s daf_butler [ 11300 ]
            Component/s pipe_base [ 10727 ]
            jbosch Jim Bosch made changes -
            Component/s pipe_tasks [ 10726 ]
            jbosch Jim Bosch made changes -
            Component/s ci_hsc_gen3 [ 17600 ]
            Component/s ip_isr [ 10730 ]
            Component/s obs_decam [ 12851 ]
            Component/s obs_lsst [ 16504 ]
            Component/s obs_subaru [ 10747 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            Hide
            jbosch Jim Bosch added a comment -

            All work was done on DM-15536; I should have closed this a while ago.

            Show
            jbosch Jim Bosch added a comment - All work was done on DM-15536 ; I should have closed this a while ago.
            fritzm Fritz Mueller made changes -
            Team Data Access and Database [ 10204 ] Architecture [ 10304 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Nate Lust, Tim Jenness
              Watchers:
              Christopher Waters, Jim Bosch, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: