Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: DM Subsystem Science, Requirements Documents
-
Epic Link:
-
Team:DM Science
Description
The DPDD tells us that "Level 1 data products are a result of difference image analysis" (§3.1), as opposed to "Level 2 data products [which] result from direct image analysis" (§4.1).
However, this definition is incorrect. Per LSE-29 (which is the highest-level document in which I can find this defined), "Level 1 Data Products [are] the result of processing of the stream of image data from the Data Collection system during the course of normal observing" (LSR-REQ-0033), while "Level 2 Data Products [are] the result of periodic processing of the entire archive of raw image data from the Data Collection system" (LSR-REQ-0036).
This isn't just mindless pedantry: I think there's an important distinction to be made here. A literal reading of the DPDD would suggest that level some 1 data products are generated during an annual data release, while some of the products of nightly processing (e.g. PVIs) are actually level 2. That's inconsistent with usage elsewhere, and I find it actively confusing. Can we fix it, please?
Attachments
Activity
The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis).
Colin Slater
added a comment - The DPDD has now been updated to replace the terms Level 1 and Level 2 with Prompt and Data Release data products, but Section 1.3 still carries over a misleading explanation of distinction between these two categories (difference vs direct image analysis). John Swinbank, I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database.
Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket.
Colin Slater
added a comment - John Swinbank , I've attempted to clear up the remaining Level 1 vs Level 2/Prompt vs DR issues lurking in the document; would you like to confirm that this now matches your understanding? Section 1.3, "Classes of LSST Data Products", has the bulk of the changes to how the two categories are differentiated. I've also used this chance to update section 3.3.5 which describes image differencing in DRP, and I've throughout the document removed references to the "Level 1"/"Level 2" Databases in favor of the PPDB/data release database.
Note that some of the oldtext/newtext red/blue text in the rendered document is from other changes already committed to master but not yet submitted to LCR, so not all of the highlighted text is specific to this ticket. Sorry for the slow turnaround on the review, and thank you for making this change: I think the document is substantially improved.
I have made a few comments on the PR. Most of these are just picking of nits; please use your own judgement as to whether or not you want to act on them.
Apart from that, I still find the statement at the beginning of §3 that “prompt data products are a result of difference image analysis” to be problematic — visit images, for example, are prompt data products that are not the result of such an analysis.
Good catch, John Swinbank, about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master.
Colin Slater
added a comment - Good catch, John Swinbank , about the intro to section 3. I missed that one, and now fixed it to be consistent with the rest of the document (at the risk of being slightly repetitive). I also implemented most of the suggestions from the PR. Merged to master. 
Lynne Jones — your understanding is basically consistent with the LSSE System Requirements (LSE-29), which I think is authoritative in this case. The issue raised by this ticket is that the DPDD (LSE-163) uses a different definition, which is confusing and needs to be fixed.
You are correct that (effectively) all nightly data products will be regenerated during annual data releases.