Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32071

DC2 Reprocessing with w_2021_40+ (gen2)

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Story Points:
      4
    • Epic Link:
    • Sprint:
      DRP S21b, DRP S22A, DRP S22B
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      Process a single tract of the DC2 dataset with the w_2021_40 + the ticket branches from DM-30284 stack using gen2 middleware. Will follow same procedure as in DM-31665 but we are setting up all branches currently in progress on DM-30284 as these are now known gen2/gen3 parity breaking issue for the coaddition stages (so this may take a few iterations as bugs there get worked out).

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            As far as gen2/gen3 parity goes, with the branches of DM-30284, we are one healthy step closer. The following plots show direct gen2/gen3 comparisons of the coadd catalogs for the previous w_2021_36 (DM-31664) run vs. this one:

            w_2021_36:

            w_2021_40:

            w_2021_36:

            w_2021_40:

            w_2021_36:

            w_2021_40:

            The percentage of matched sources has also increased from 82-92% for w_2021_36 to 89-97% for w_2021_40.

            I believe I have tracked the final differences down to (mild) sensitivities of the makeWarp task to the input order of detectors and of the scarlet deblender to the input order of bands. The fix, which is to sort the detector/band order in the task itself so that the ordering sent to the ordering in the task execution is guaranteed regardless of input ordering, does involve a change to gen3, so this can only be confirmed in the context of our weekly processing once the fixes in DM-30284 have been merged.
             

            Show
            lauren Lauren MacArthur added a comment - As far as gen2/gen3 parity goes, with the branches of DM-30284 , we are one healthy step closer. The following plots show direct gen2/gen3 comparisons of the coadd catalogs for the previous w_2021_36 ( DM-31664 ) run vs. this one: w_2021_36: w_2021_40: w_2021_36: w_2021_40: w_2021_36: w_2021_40: The percentage of matched sources has also increased from 82-92% for w_2021_36 to 89-97% for  w_2021_40 . I believe I have tracked the final differences down to (mild) sensitivities of the makeWarp task to the input order of detectors and of the scarlet deblender to the input order of bands. The fix, which is to sort the detector/band order in the task itself so that the ordering sent to the ordering in the task execution is guaranteed regardless of input ordering, does involve a change to gen3, so this can only be confirmed in the context of our weekly processing once the fixes in DM-30284 have been merged.  
            Hide
            lauren Lauren MacArthur added a comment - - edited

            The pipe_analysis gen2 plots are linked here and the gen3 & gen2 vs. gen3 plots are here.

            The verifyDrp plots are linked here.

            Run logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/DM-32071/logs and linked here.

            QA logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/DM-32071/qaLogs and linked here.

            Metrics were dispatched and ingested. They can be seen on the dashboard titled "DRP metrics monthly for DC2 (Gen2)" at https://chronograf-demo.lsst.codes/sources/2/dashboards. They all look very similar to the w_2021_36 run.

            Show
            lauren Lauren MacArthur added a comment - - edited The pipe_analysis gen2 plots are linked here and the gen3 & gen2 vs. gen3 plots are here . The verifyDrp plots are linked here . Run logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/ DM-32071 /logs and linked here . QA logs are in /datasets/DC2/repoRun2.2i/rerun/w_2021_40/ DM-32071 /qaLogs and linked here . Metrics were dispatched and ingested. They can be seen on the dashboard titled "DRP metrics monthly for DC2 (Gen2)" at https://chronograf-demo.lsst.codes/sources/2/dashboards . They all look very similar to the w_2021_36 run.
            Hide
            lauren Lauren MacArthur added a comment -

            Catching up...another DC2 run for you to take a gander at.

            Show
            lauren Lauren MacArthur added a comment - Catching up...another DC2 run for you to take a gander at.

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Lauren MacArthur, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.