Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26364

Science Pipelines Release v20 Acceptance Test Campaign

    Details

    • Story Points:
      8
    • Epic Link:
    • Team:
      DM Science
    • Urgent?:
      No

      Description

      Carry out acceptance testing of v20.0 Science Pipelines Release.

        Attachments

          Issue Links

            Activity

            Hide
            jcarlin Jeffrey Carlin added a comment -

            DMTR-261 is at this link.

            Show
            jcarlin Jeffrey Carlin added a comment - DMTR-261 is at this link .
            Hide
            gcomoretto Gabriele Comoretto added a comment -

            For extended comments on the content, please use the PR:

            https://github.com/lsst-dm/DMTR-261/pull/2

            Show
            gcomoretto Gabriele Comoretto added a comment - For extended comments on the content, please use the PR: https://github.com/lsst-dm/DMTR-261/pull/2
            Hide
            lguy Leanne Guy added a comment -

            Nice work and great too see 14 newly verified test cases!

            For the recommended improvements, we should create tickets to schedule this work.

            Minor comments to address then this is done:

            • P1: This Acceptance Test campaign aims to verify a subset of DMSR (LSE-61) requirements related ...
              > This Acceptance Test campaign aims to verify a subset of DM subsystem requirements (LSE-61) related to ....
            • P1 readiness for LSST Operations => readiness for LSST commissioning .
              (commissioning verifies readiness for operations)
            • P1: The tests will be performed using the HSC-RC2 dataset (as defined in DM-11345 ).
              Reference should be DMTN-091: Test Datasets for Scientific Performance Monitoring. If DMTN-091 is not correct, we must update it.
            • P45: 5.1.3.11 LVV-T1757 – "Note that because the science pipelines are still in development, the actual resulting values for these metrics are not important." -> I would not phrase it as "not important", rather, that the values obtained are not the focus of this test and that the actual performance will be verified during commissioning on LSST data using this code. We do however expect to get reasonable results with precursor data and if, for example, we were seeing mag-level numbers for PA1 on HSC, we'd be looking for bugs.
            • Likewise for LVV-T1758, LVV-T1756, LVV-T1759
            • P 7, 56: A reference has not rendered "and ingested as part of the Operations Rehearsal detailed in ? : “LDM-503-11a: ComCam
              OPS Readiness Test Plan and Report”
            Show
            lguy Leanne Guy added a comment - Nice work and great too see 14 newly verified test cases! For the recommended improvements, we should create tickets to schedule this work. Minor comments to address then this is done: P1: This Acceptance Test campaign aims to verify a subset of DMSR (LSE-61) requirements related ... > This Acceptance Test campaign aims to verify a subset of DM subsystem requirements (LSE-61) related to .... P1 readiness for LSST Operations => readiness for LSST commissioning . (commissioning verifies readiness for operations) P1: The tests will be performed using the HSC-RC2 dataset (as defined in DM-11345 ). Reference should be DMTN-091: Test Datasets for Scientific Performance Monitoring. If DMTN-091 is not correct, we must update it. P45: 5.1.3.11 LVV-T1757 – "Note that because the science pipelines are still in development, the actual resulting values for these metrics are not important." -> I would not phrase it as "not important", rather, that the values obtained are not the focus of this test and that the actual performance will be verified during commissioning on LSST data using this code. We do however expect to get reasonable results with precursor data and if, for example, we were seeing mag-level numbers for PA1 on HSC, we'd be looking for bugs. Likewise for LVV-T1758, LVV-T1756, LVV-T1759 P 7, 56: A reference has not rendered "and ingested as part of the Operations Rehearsal detailed in ? : “LDM-503-11a: ComCam OPS Readiness Test Plan and Report”
            Hide
            jcarlin Jeffrey Carlin added a comment -

            I have updated the document(s) to address these comments, and created a ticket (DM-25786) to address the first of the recommended improvements. The second suggested improvement is being addressed by the new metrics tooling that is being developed, and the third recommendation (update LVV-T125 to test fake source insertion more carefully) has a ticket assigned to it (DM-25787).

            If the links to references render properly, then DMTR-261 should be complete.

            Show
            jcarlin Jeffrey Carlin added a comment - I have updated the document(s) to address these comments, and created a ticket ( DM-25786 ) to address the first of the recommended improvements. The second suggested improvement is being addressed by the new metrics tooling that is being developed, and the third recommendation (update LVV-T125 to test fake source insertion more carefully) has a ticket assigned to it ( DM-25787 ). If the links to references render properly, then DMTR-261 should be complete.
            Hide
            jcarlin Jeffrey Carlin added a comment - - edited

            Final version of DMTR-261 is available in Docushare.

            Show
            jcarlin Jeffrey Carlin added a comment - - edited Final version of DMTR-261 is available in Docushare.

              People

              • Assignee:
                jcarlin Jeffrey Carlin
                Reporter:
                jcarlin Jeffrey Carlin
                Reviewers:
                Leanne Guy
                Watchers:
                Gabriele Comoretto, Jeffrey Carlin, Leanne Guy
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: