Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23445

Move visit definition out of raw ingest for Gen3

    Details

    • Templates:
    • Story Points:
      0
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      At present, RawIngestTask is responsible for adding three different types of dimension records:

      • exposure
      • visit
      • visit_detector_region

      We should move the latter two to a separate script that is run after RawIngestTask and (at least at present) only after all associated raws for a visit have been ingested. That will open the door to (later) being able to define different visit definitions for the same exposure (DM-15536).

      This will not on its own allow individual raws to be ingested incrementally - RawIngestTask still needs all raws for an exposure to be present in a single invocation so we can add the exposure dimension record once without needing to see if it already exists in the Registry. That could be fixed by using Database.sync in a new high-level counterpart to Registry.insertDimensionData that does perform such a check, but we'll first need to figure out how to handle transactions in ingest - Database.sync is not and probably cannot be part of a larger transaction.

      But even that is an easier problem to solve than the corresponding one for visits, because (in addition to the synchronization problem) the visit dimension record needs to contain a region that is currently computed from the regions of all of the detectors that went into it (but could be computed from cameraGeom in the future).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Nate Lust, Tim Jenness
                Watchers:
                Christopher Waters, Jim Bosch, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel