Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10540

DPDD incorrectly defines "level 1" and "level 2"

    Details

      Description

      The DPDD tells us that "Level 1 data products are a result of difference image analysis" (§3.1), as opposed to "Level 2 data products [which] result from direct image analysis" (§4.1).

      However, this definition is incorrect. Per LSE-29 (which is the highest-level document in which I can find this defined), "Level 1 Data Products [are] the result of processing of the stream of image data from the Data Collection system during the course of normal observing" (LSR-REQ-0033), while "Level 2 Data Products [are] the result of periodic processing of the entire archive of raw image data from the Data Collection system" (LSR-REQ-0036).

      This isn't just mindless pedantry: I think there's an important distinction to be made here. A literal reading of the DPDD would suggest that level some 1 data products are generated during an annual data release, while some of the products of nightly processing (e.g. PVIs) are actually level 2. That's inconsistent with usage elsewhere, and I find it actively confusing. Can we fix it, please?

        Attachments

          Activity

          Hide
          ctslater Colin Slater added a comment -

          Good catch, John Swinbank, about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master.

          Show
          ctslater Colin Slater added a comment - Good catch, John Swinbank , about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master.
          Hide
          swinbank John Swinbank added a comment -

          Sorry for the slow turnaround on the review, and thank you for making this change: I think the document is substantially improved.

          I have made a few comments on the PR. Most of these are just picking of nits; please use your own judgement as to whether or not you want to act on them.

          Apart from that, I still find the statement at the beginning of §3 that “prompt data products are a result of difference image analysis” to be problematic — visit images, for example, are prompt data products that are not the result of such an analysis.

          Show
          swinbank John Swinbank added a comment - Sorry for the slow turnaround on the review, and thank you for making this change: I think the document is substantially improved. I have made a few comments on the PR. Most of these are just picking of nits; please use your own judgement as to whether or not you want to act on them. Apart from that, I still find the statement at the beginning of §3 that “prompt data products are a result of difference image analysis” to be problematic — visit images, for example, are prompt data products that are not the result of such an analysis.
          Hide
          ctslater Colin Slater added a comment -

          John Swinbank, I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database.

          Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket.

          Show
          ctslater Colin Slater added a comment - John Swinbank , I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database. Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket.
          Hide
          ctslater Colin Slater added a comment -

          The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis).

          Show
          ctslater Colin Slater added a comment - The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis).
          Hide
          swinbank John Swinbank added a comment -

          Lynne Jones — your understanding is basically consistent with the LSSE System Requirements (LSE-29), which I think is authoritative in this case. The issue raised by this ticket is that the DPDD (LSE-163) uses a different definition, which is confusing and needs to be fixed.

          You are correct that (effectively) all nightly data products will be regenerated during annual data releases.

          Show
          swinbank John Swinbank added a comment - Lynne Jones — your understanding is basically consistent with the LSSE System Requirements (LSE-29), which I think is authoritative in this case. The issue raised by this ticket is that the DPDD (LSE-163) uses a different definition, which is confusing and needs to be fixed. You are correct that (effectively) all nightly data products will be regenerated during annual data releases.

            People

            • Assignee:
              ctslater Colin Slater
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Swinbank
              Watchers:
              Colin Slater, Eric Bellm, Gregory Dubois-Felsmann, John Swinbank, Kian-Tat Lim, Leanne Guy, Lynne Jones, Tim Jenness, Wil O'Mullane
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Summary Panel