Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15536

Add level of indirection in defining Visits from Exposures

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      In discussions with Robert Lupton, we decided that it would be best for multiple mutually-inconsistent associations of exposures (snaps) into visits should be permitted in the same Gen3 data repository.  We cannot assume that we will be able to do that association in advance and not want to revisit it in later reprocessing.

      On the database side, I think we'd need to add a field to the visit table indicating which of several named systems the visit belongs to, while ensuring that visit IDs are unique across all systems.  We'd select exactly one system via configuration to be used by any butler client, though we'd need to think about when we'd have to be able to refer to other systems instead of relying on that selection (e.g. when interacting with datasets produced on some other system).  We'll also need to move the exposure.visit field into a separate join table.

      The way information about visits will be transmitted from the LSST hardware is via a "GROUP" header card; to make use of that, we should:

      • propagate that through ObservationInfo into the exposure table during raw ingest;
      • remove visit definition (and all of the stuff dealing with visit regions) from raw ingest;
      • add a new task to be run after ingest that creates visits (with regions, where appropriate) from groups of exposures with the same GROUP value.  That would involve generating visit IDs and names from GROUP via some algorithm.

      For instruments that want visits to be 1-1 with exposures, we'll just define GROUP accordingly.  If we ever want to define a set of visits that isn't strictly derived from GROUP, we can write an alternate version of the task that populates the same tables via some other means.

        Attachments

          Issue Links

            Activity

            jbosch Jim Bosch created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue is triggered by RFC-484 [ RFC-484 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 17995 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20471 ]
            jbosch Jim Bosch made changes -
            Description In discussions with [~rhl], we decided that it would be best for multiple mutually-inconsistent associations of Exposures (snaps) into Visits should be permitted in the same Gen3 data repository.  We cannot assume that we will be able to do that association in advance and not want to revisit it in later reprocessing.

            This will entail adding a new DataUnit (proposed name is "SnapSystem") that defines a named association of Exposure into Visits, along with a join table that relates Exposure, Visit, and SnapSystem.  The visit field will be removed from Exposure itself, and some metadata from Visit will be moved or copied to Exposure (including at least the region).
            In discussions with [~rhl], we decided that it would be best for multiple mutually-inconsistent associations of exposures (snaps) into visits should be permitted in the same Gen3 data repository.  We cannot assume that we will be able to do that association in advance and not want to revisit it in later reprocessing.

            On the database side, I think we'd need to add a field to the visit table indicating which of several named systems the visit belongs to, while ensuring that visit IDs are unique across all systems.  We'd select exactly one system via configuration to be used by any butler client, though we'd need to think about when we'd have to be able to refer to other systems instead of relying on that selection (e.g. when interacting with datasets produced on some other system).  We'll also need to move the {{exposure.visit}} field into a separate join table.

            The way information about visits will be transmitted from the LSST hardware is via a "GROUP" header card; to make use of that, we should:
             * propagate that through ObservationInfo into the exposure table during raw ingest;
             * remove visit definition (and all of the stuff dealing with visit regions) from raw ingest;
             * add a new task to be run after ingest that creates visits (with regions, where appropriate) from groups of exposures with the same GROUP value.  That would involve generating visit IDs and names from GROUP via some algorithm.

            For instruments that want visits to be 1-1 with exposures, we'll just define GROUP accordingly.  If we ever want to define a set of visits that isn't strictly derived from GROUP, we can write an alternate version of the task that populates the same tables via some other means.
            jbosch Jim Bosch made changes -
            Labels gen3-middleware gen2-deprecation-blocker gen3-middleware
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-23445 [ DM-23445 ]
            jbosch Jim Bosch made changes -
            Link This issue is blocked by DM-23445 [ DM-23445 ]
            jbosch Jim Bosch made changes -
            Link This issue blocks DM-23445 [ DM-23445 ]
            jbosch Jim Bosch made changes -
            Story Points 4 2
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-23208 [ DM-23208 ]
            tjenness Tim Jenness made changes -
            Labels gen2-deprecation-blocker gen3-middleware gen2-deprecation-blocker gen3-middleware gen3-registry-incompatibility
            jbosch Jim Bosch made changes -
            Story Points 2 8
            jbosch Jim Bosch made changes -
            Reviewers Tim Jenness [ tjenness ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            tjenness Tim Jenness made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-23737 [ 431393 ]
            swinbank John Swinbank made changes -
            Urgent? off
            tjenness Tim Jenness made changes -
            Link This issue is duplicated by DM-23843 [ DM-23843 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Tim Jenness
              Watchers:
              Gregory Dubois-Felsmann, Jim Bosch, Michelle Gower, Nate Lust, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.