Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
As per the discussion on DM-30647, the detailed outputs from the postprocess step which creates various summary/aggregated/DPDD-ified tables (most of which are persisted as parquet). This includes at least the sourceTable and sourceTable_visit datasets, but there may be a few more to check in the next processing run (e.g. I think CcdVisitTable and visitTable have been recently added, but it may be that we only ever produce these with gen3).
Attachments
Issue Links
- relates to
-
- Done
Activity
As per my comment in section 2. of https://confluence.lsstcorp.org/display/DM/Gen2+Middleware+Removal+Planning:
Oh, one final note: I have only done some spot-checking comparing the rolled up and DPDD-ified parquet tables in detail. The per-detector/per-patch non DPDD versions are already effectively tested since those are used in the plotting scripts. So far, the only differences I've found are a few different column entries;
objectTables: the Gen3 has one column entry that the Gen2 does not: {'skymap'}
sourceTables: Gen3 has {'band', 'instrument', 'physical_filter', 'visit_system'}
Gen2 has {'detectorName', 'expId', 'raftName', 'run'}
this check was not done in an exhaustive fashion, but I was convinced by my spot checking that this was not necessary. Just to be sure, I’m putting this in review to Jim Bosch to give him a chance to request a more extensive comparison (assuming the latest gen2 runs actually got transferred to the new USDF at SLAC 
)
Is this relevant any more?