Fix Version/s: None
Sprint:DRP S21b, DRP S22A, DRP S22B
Team:Data Release Production
Process a single tract of the DC2 dataset with the w_2021_40 + the ticket branches from
DM-30284 stack using gen2 middleware. Will follow same procedure as in DM-31665 but we are setting up all branches currently in progress on DM-30284 as these are now known gen2/gen3 parity breaking issue for the coaddition stages (so this may take a few iterations as bugs there get worked out).
The pipe_analysis gen2 plots are linked here and the gen3 & gen2 vs. gen3 plots are here.
The verifyDrp plots are linked here.
Run logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/
DM-32071/logs and linked here.
QA logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/
DM-32071/qaLogs and linked here.
Metrics were dispatched and ingested. They can be seen on the dashboard titled "DRP metrics monthly for DC2 (Gen2)" at https://chronograf-demo.lsst.codes/sources/2/dashboards. They all look very similar to the w_2021_36 run.
As far as gen2/gen3 parity goes, with the branches of
DM-30284, we are one healthy step closer. The following plots show direct gen2/gen3 comparisons of the coadd catalogs for the previous w_2021_36 ( DM-31664) run vs. this one:
The percentage of matched sources has also increased from 82-92% for w_2021_36 to 89-97% for w_2021_40.
I believe I have tracked the final differences down to (mild) sensitivities of the makeWarp task to the input order of detectors and of the scarlet deblender to the input order of bands. The fix, which is to sort the detector/band order in the task itself so that the ordering sent to the ordering in the task execution is guaranteed regardless of input ordering, does involve a change to gen3, so this can only be confirmed in the context of our weekly processing once the fixes in
DM-30284have been merged.