Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23445

Move visit definition out of raw ingest for Gen3


    • Templates:
    • Story Points:
    • Team:
      Data Access and Database
    • Urgent?:


      At present, RawIngestTask is responsible for adding three different types of dimension records:

      • exposure
      • visit
      • visit_detector_region

      We should move the latter two to a separate script that is run after RawIngestTask and (at least at present) only after all associated raws for a visit have been ingested. That will open the door to (later) being able to define different visit definitions for the same exposure (DM-15536).

      This will not on its own allow individual raws to be ingested incrementally - RawIngestTask still needs all raws for an exposure to be present in a single invocation so we can add the exposure dimension record once without needing to see if it already exists in the Registry. That could be fixed by using Database.sync in a new high-level counterpart to Registry.insertDimensionData that does perform such a check, but we'll first need to figure out how to handle transactions in ingest - Database.sync is not and probably cannot be part of a larger transaction.

      But even that is an easier problem to solve than the corresponding one for visits, because (in addition to the synchronization problem) the visit dimension record needs to contain a region that is currently computed from the regions of all of the detectors that went into it (but could be computed from cameraGeom in the future).


          Issue Links



              • Assignee:
                jbosch Jim Bosch
                jbosch Jim Bosch
                Nate Lust, Tim Jenness
                Christopher Waters, Jim Bosch, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created:

                  Summary Panel