Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30817

Compare the gen2 vs. gen3 visit-level post-processing data products

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Epic Link:
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As per the discussion on DM-30647, the detailed outputs from the postprocess step which creates various summary/aggregated/DPDD-ified tables (most of which are persisted as parquet).  This includes at least the sourceTable and sourceTable_visit datasets, but there may be a few more to check in the next processing run (e.g. I think CcdVisitTable and visitTable have been recently added, but it may be that we only ever produce these with gen3).

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30647 Compare the data products of the gen2 vs. gen3 w_2021_22 RC2 runs up to Single Frame Processing

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue relates to DM-30647 [ DM-30647 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30472 [ 509197 ] DM-30540 [ 511198 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30540 [ 511198 ] DM-30547 [ 511213 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30547 [ 511213 ] DM-30555 [ 511221 ]
            yusra Yusra AlSayyad made changes -
            Epic Link DM-30555 [ 511221 ] DM-32157 [ 779840 ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Is this relevant any more?

            Show
            tjenness Tim Jenness added a comment - Is this relevant any more?
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            As per my comment in section 2. of https://confluence.lsstcorp.org/display/DM/Gen2+Middleware+Removal+Planning:

            Oh, one final note: I have only done some spot-checking comparing the rolled up and DPDD-ified parquet tables in detail.  The per-detector/per-patch non DPDD versions are already effectively tested since those are used in the plotting scripts.  So far, the only differences I've found are a few different column entries;

            objectTables: the Gen3 has one column entry that the Gen2 does not: {'skymap'}

            sourceTables: Gen3 has {'band', 'instrument', 'physical_filter', 'visit_system'}

                                      Gen2 has {'detectorName', 'expId', 'raftName', 'run'}

            this check was not done in an exhaustive fashion, but I was convinced by my spot checking that this was not necessary. Just to be sure, I’m putting this in review to Jim Bosch to give him a chance to request a more extensive comparison (assuming the latest gen2 runs actually got transferred to the new USDF at SLAC )

            Show
            lauren Lauren MacArthur added a comment - As per my comment in section 2. of https://confluence.lsstcorp.org/display/DM/Gen2+Middleware+Removal+Planning : Oh, one final note: I have only done some spot-checking comparing the rolled up and DPDD-ified parquet tables in detail.  The per-detector/per-patch non DPDD versions are already effectively tested since those are used in the plotting scripts.  So far, the only differences I've found are a few different column entries; objectTables: the Gen3 has one column entry that the Gen2 does not: {'skymap'} sourceTables: Gen3 has {'band', 'instrument', 'physical_filter', 'visit_system'}                           Gen2 has {'detectorName', 'expId', 'raftName', 'run'} this check was not done in an exhaustive fashion, but I was convinced by my spot checking that this was not necessary. Just to be sure, I’m putting this in review to Jim Bosch to give him a chance to request a more extensive comparison (assuming the latest gen2 runs actually got transferred to the new USDF at SLAC )
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            Reviewers Jim Bosch [ jbosch ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            No need for more work on this whatsoever.

            Show
            jbosch Jim Bosch added a comment - No need for more work on this whatsoever.
            jbosch Jim Bosch made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Story Points 1

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Lauren MacArthur, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.