Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23445

Move visit definition out of raw ingest for Gen3

    XMLWordPrintable

    Details

    • Story Points:
      0
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      At present, RawIngestTask is responsible for adding three different types of dimension records:

      • exposure
      • visit
      • visit_detector_region

      We should move the latter two to a separate script that is run after RawIngestTask and (at least at present) only after all associated raws for a visit have been ingested. That will open the door to (later) being able to define different visit definitions for the same exposure (DM-15536).

      This will not on its own allow individual raws to be ingested incrementally - RawIngestTask still needs all raws for an exposure to be present in a single invocation so we can add the exposure dimension record once without needing to see if it already exists in the Registry. That could be fixed by using Database.sync in a new high-level counterpart to Registry.insertDimensionData that does perform such a check, but we'll first need to figure out how to handle transactions in ingest - Database.sync is not and probably cannot be part of a larger transaction.

      But even that is an easier problem to solve than the corresponding one for visits, because (in addition to the synchronization problem) the visit dimension record needs to contain a region that is currently computed from the regions of all of the detectors that went into it (but could be computed from cameraGeom in the future).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Nate Lust, Tim Jenness
              Watchers:
              Christopher Waters, Jim Bosch, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: