Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10540

DPDD incorrectly defines "level 1" and "level 2"

    Details

      Description

      The DPDD tells us that "Level 1 data products are a result of difference image analysis" (§3.1), as opposed to "Level 2 data products [which] result from direct image analysis" (§4.1).

      However, this definition is incorrect. Per LSE-29 (which is the highest-level document in which I can find this defined), "Level 1 Data Products [are] the result of processing of the stream of image data from the Data Collection system during the course of normal observing" (LSR-REQ-0033), while "Level 2 Data Products [are] the result of periodic processing of the entire archive of raw image data from the Data Collection system" (LSR-REQ-0036).

      This isn't just mindless pedantry: I think there's an important distinction to be made here. A literal reading of the DPDD would suggest that level some 1 data products are generated during an annual data release, while some of the products of nightly processing (e.g. PVIs) are actually level 2. That's inconsistent with usage elsewhere, and I find it actively confusing. Can we fix it, please?

        Attachments

          Activity

          Hide
          mjuric Mario Juric added a comment -

          I agree, this is a DPDD bug.

          Show
          mjuric Mario Juric added a comment - I agree, this is a DPDD bug.
          Hide
          tjenness Tim Jenness added a comment -

          And I propagated the confusion into DMS-REQ-0325:

          1.1.1.1 Regenerating L1 Data Products During Data Release Processing

          Specification: During Data Release Processing, all the Level 1 data products shall be regenerated using the current best algorithms.

          In the LSE-29 definition L1 data products are entirely transient and disappear when caches expire. Common usage more closely matches the DPDD definition where L1 data products are the product types that we generate on the night. Everyone agrees that L2 data products are the ones we generate annually, except many people forget that that includes the "L1 data products" (hence my adding a requirement making that explicit).

          Show
          tjenness Tim Jenness added a comment - And I propagated the confusion into DMS-REQ-0325: 1.1.1.1 Regenerating L1 Data Products During Data Release Processing Specification: During Data Release Processing, all the Level 1 data products shall be regenerated using the current best algorithms. In the LSE-29 definition L1 data products are entirely transient and disappear when caches expire. Common usage more closely matches the DPDD definition where L1 data products are the product types that we generate on the night. Everyone agrees that L2 data products are the ones we generate annually, except many people forget that that includes the "L1 data products" (hence my adding a requirement making that explicit).
          Hide
          ktl Kian-Tat Lim added a comment -

          I think Tim's usage is reasonable; all of the Level 1 data products are regenerated during the DRP and the results are L2 data products (or L2 versions of the L1 products). It's definitely not the case that only results from DIA are L1 data products.

          Show
          ktl Kian-Tat Lim added a comment - I think Tim's usage is reasonable; all of the Level 1 data products are regenerated during the DRP and the results are L2 data products (or L2 versions of the L1 products). It's definitely not the case that only results from DIA are L1 data products.
          Hide
          tjenness Tim Jenness added a comment -

          Furthermore:

          LSR-REQ-0049

          Requirement: The LSST system shall archive all generated Level 1, Level 2, and Calibration Data Products, or provide services to reconstruct any given data product on demand.

          which is implying that Level 1 data products are a thing and that they are archived.

          Show
          tjenness Tim Jenness added a comment - Furthermore: LSR-REQ-0049 Requirement: The LSST system shall archive all generated Level 1, Level 2, and Calibration Data Products, or provide services to reconstruct any given data product on demand. which is implying that Level 1 data products are a thing and that they are archived.
          Hide
          gpdf Gregory Dubois-Felsmann added a comment -

          Is there still an issue here, or can this be closed?

          Show
          gpdf Gregory Dubois-Felsmann added a comment - Is there still an issue here, or can this be closed?
          Hide
          swinbank John Swinbank added a comment -

          The DPDD hasn't been changed as far as I know. It's still wrong.

          Show
          swinbank John Swinbank added a comment - The DPDD hasn't been changed as far as I know. It's still wrong.
          Hide
          mjuric Mario Juric added a comment -

          Correct, this still needs to be fixed. I think there's broad agreement here on what to do, but we didn't get to it yet. I hope to schedule it sometime this winter.

          (the more difficult bit will be to get everyone to consistently correctly use the terms.)

          Show
          mjuric Mario Juric added a comment - Correct, this still needs to be fixed. I think there's broad agreement here on what to do, but we didn't get to it yet. I hope to schedule it sometime this winter. (the more difficult bit will be to get everyone to consistently correctly use the terms.)
          Hide
          ljones Lynne Jones added a comment - - edited

          This ticket has confused me. My understanding of this was more like

          • "level 1" == daily/nightly processing results
          • "level 2" == yearly/DRP processing results

          And that DIASources and DIAObjects and SSObjects would be generated nightly/daily via difference imaging, source association / alert production, and MOPS (creating the 'level 1 MOPS database' of SSObjects, which is 'live' and changes daily, with updated orbits, etc).

          And that during DRP, DIASources would be generated again using difference imaging (potentially with different parameters than the nightly processing), DIAObjects would also be regenerated , but definitely MOPS would be re-run (creating the "level 2 / DRP MOPS database of SSObjects" which is static and could be entirely different from the equivalently dated level 1 db), and then Sources and Objects would also be measured.

          Is that not correct? Are DIAsources not going to be regenerated during DRP?

          Show
          ljones Lynne Jones added a comment - - edited This ticket has confused me. My understanding of this was more like "level 1" == daily/nightly processing results "level 2" == yearly/DRP processing results And that DIASources and DIAObjects and SSObjects would be generated nightly/daily via difference imaging, source association / alert production, and MOPS (creating the 'level 1 MOPS database' of SSObjects, which is 'live' and changes daily, with updated orbits, etc). And that during DRP, DIASources would be generated again using difference imaging (potentially with different parameters than the nightly processing), DIAObjects would also be regenerated , but definitely MOPS would be re-run (creating the "level 2 / DRP MOPS database of SSObjects" which is static and could be entirely different from the equivalently dated level 1 db), and then Sources and Objects would also be measured. Is that not correct? Are DIAsources not going to be regenerated during DRP?
          Hide
          swinbank John Swinbank added a comment -

          Lynne Jones — your understanding is basically consistent with the LSSE System Requirements (LSE-29), which I think is authoritative in this case. The issue raised by this ticket is that the DPDD (LSE-163) uses a different definition, which is confusing and needs to be fixed.

          You are correct that (effectively) all nightly data products will be regenerated during annual data releases.

          Show
          swinbank John Swinbank added a comment - Lynne Jones — your understanding is basically consistent with the LSSE System Requirements (LSE-29), which I think is authoritative in this case. The issue raised by this ticket is that the DPDD (LSE-163) uses a different definition, which is confusing and needs to be fixed. You are correct that (effectively) all nightly data products will be regenerated during annual data releases.
          Hide
          ctslater Colin Slater added a comment -

          The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis).

          Show
          ctslater Colin Slater added a comment - The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis).
          Hide
          ctslater Colin Slater added a comment -

          John Swinbank, I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database.

          Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket.

          Show
          ctslater Colin Slater added a comment - John Swinbank , I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database. Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket.
          Hide
          swinbank John Swinbank added a comment -

          Sorry for the slow turnaround on the review, and thank you for making this change: I think the document is substantially improved.

          I have made a few comments on the PR. Most of these are just picking of nits; please use your own judgement as to whether or not you want to act on them.

          Apart from that, I still find the statement at the beginning of §3 that “prompt data products are a result of difference image analysis” to be problematic — visit images, for example, are prompt data products that are not the result of such an analysis.

          Show
          swinbank John Swinbank added a comment - Sorry for the slow turnaround on the review, and thank you for making this change: I think the document is substantially improved. I have made a few comments on the PR. Most of these are just picking of nits; please use your own judgement as to whether or not you want to act on them. Apart from that, I still find the statement at the beginning of §3 that “prompt data products are a result of difference image analysis” to be problematic — visit images, for example, are prompt data products that are not the result of such an analysis.
          Hide
          ctslater Colin Slater added a comment -

          Good catch, John Swinbank, about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master.

          Show
          ctslater Colin Slater added a comment - Good catch, John Swinbank , about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master.

            People

            • Assignee:
              ctslater Colin Slater
              Reporter:
              swinbank John Swinbank
              Reviewers:
              John Swinbank
              Watchers:
              Colin Slater, Eric Bellm, Gregory Dubois-Felsmann, John Swinbank, Kian-Tat Lim, Leanne Guy, Lynne Jones, Tim Jenness, Wil O'Mullane
            • Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Summary Panel