Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30817

Compare the gen2 vs. gen3 visit-level post-processing data products

    XMLWordPrintable

Details

    • 1
    • Data Release Production
    • No

    Description

      As per the discussion on DM-30647, the detailed outputs from the postprocess step which creates various summary/aggregated/DPDD-ified tables (most of which are persisted as parquet).  This includes at least the sourceTable and sourceTable_visit datasets, but there may be a few more to check in the next processing run (e.g. I think CcdVisitTable and visitTable have been recently added, but it may be that we only ever produce these with gen3).

      Attachments

        Issue Links

          Activity

            tjenness Tim Jenness added a comment -

            Is this relevant any more?

            tjenness Tim Jenness added a comment - Is this relevant any more?

            As per my comment in section 2. of https://confluence.lsstcorp.org/display/DM/Gen2+Middleware+Removal+Planning:

            Oh, one final note: I have only done some spot-checking comparing the rolled up and DPDD-ified parquet tables in detail.  The per-detector/per-patch non DPDD versions are already effectively tested since those are used in the plotting scripts.  So far, the only differences I've found are a few different column entries;

            objectTables: the Gen3 has one column entry that the Gen2 does not: {'skymap'}

            sourceTables: Gen3 has {'band', 'instrument', 'physical_filter', 'visit_system'}

                                      Gen2 has {'detectorName', 'expId', 'raftName', 'run'}

            this check was not done in an exhaustive fashion, but I was convinced by my spot checking that this was not necessary. Just to be sure, I’m putting this in review to jbosch to give him a chance to request a more extensive comparison (assuming the latest gen2 runs actually got transferred to the new USDF at SLAC )

            lauren Lauren MacArthur added a comment - As per my comment in section 2. of https://confluence.lsstcorp.org/display/DM/Gen2+Middleware+Removal+Planning : Oh, one final note: I have only done some spot-checking comparing the rolled up and DPDD-ified parquet tables in detail.  The per-detector/per-patch non DPDD versions are already effectively tested since those are used in the plotting scripts.  So far, the only differences I've found are a few different column entries; objectTables: the Gen3 has one column entry that the Gen2 does not: {'skymap'} sourceTables: Gen3 has {'band', 'instrument', 'physical_filter', 'visit_system'}                           Gen2 has {'detectorName', 'expId', 'raftName', 'run'} this check was not done in an exhaustive fashion, but I was convinced by my spot checking that this was not necessary. Just to be sure, I’m putting this in review to jbosch to give him a chance to request a more extensive comparison (assuming the latest gen2 runs actually got transferred to the new USDF at SLAC )
            jbosch Jim Bosch added a comment -

            No need for more work on this whatsoever.

            jbosch Jim Bosch added a comment - No need for more work on this whatsoever.

            People

              lauren Lauren MacArthur
              lauren Lauren MacArthur
              Jim Bosch
              Jim Bosch, Lauren MacArthur, Tim Jenness, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.