Description
The DPDD describes (most completely in Section 3.2.1) the execution of forced and precovery photomery during prompt processing. Footnote 26 says that the DPDD treats them as "just" DIASources that may be flagged appropriately in the DIASource table or stored in a separate table. This statement creates some ambiguity I'd like to resolve.
Because DIAForcedSources and DIASources have different semantic meanings, I'd first like to propose that we distinguish them explicitly in the DPDD (even if the actual storage of DIAForcedSources turns out to be as a flagged subset within the DIASource table). This would eliminate footnote 26, which is itself inconsistent with the statement in Section 3.3.1 that the DIASource table is "a table of sources detected at SNR > 5 on difference images." In my scan of the rest of the DPDD, actual usage of "DIASource" appears to be restricted to "unforced" DIASources, contra footnote 26.
I propose that we further create a DIAForcedSources table that clarifies which columns will be present in that table. I believe these should be at least:
diaForcedSourceId, ccdVisitId, diaObjectId, psFlux, psFluxErr (which is missing from DIASource!), flags.
I also think we should include totFlux and totFluxErr (forced flux on the PVI): these are required to reconstruct the DC lightcurve of variable stars when the difference image magnitude is near that of the template.
I also suggest we state explicitly that any existing DIAForcedSources already in the PPDB will be included in alert packets with the 12-month DIASource history. When available, these measurements are obviously more valuable to end-users than the upper limits provided by RFC-348.
Finally, I suggest that we clarify that DIAObject timeseries features are computed on DIASources only and not DIAForcedSources (even if present), since the latter have complex sampling histories related to the precovery window and in many cases will be nondetections.
Attachments
Issue Links
Activity
I support this whole set of clarifications.
The one other item I can think to add to the schema is the ra and dec that was input to the force photometry. Since the centroid of the DIAObject is updated each time a new DIASource is matched, the positions of a set of DIAForcedSource records won't necessarily be constant. I could imagine scenarios where one wants to know if variations in flux are due to that center wandering around. The position would otherwise need to be extracted from matching each DIAFS record to the specific DIAObject record that was valid at the time. (Though I can't say whether this is really worth the cost of two doubles x 10s of billions of records to avoid that complicated query)
Colin Slater
added a comment - I support this whole set of clarifications.
The one other item I can think to add to the schema is the ra and dec that was input to the force photometry. Since the centroid of the DIAObject is updated each time a new DIASource is matched, the positions of a set of DIAForcedSource records won't necessarily be constant. I could imagine scenarios where one wants to know if variations in flux are due to that center wandering around. The position would otherwise need to be extracted from matching each DIAFS record to the specific DIAObject record that was valid at the time. (Though I can't say whether this is really worth the cost of two doubles x 10s of billions of records to avoid that complicated query) Created LCR-1459: https://project.lsst.org/groups/ccb/node/2655
Colin Slater and I discussed the potential inclusion of the DiaObject ra and dec in the DiaForcedSourceTable. We agreed that the tradeoff between extra storage and performing a DiaForcedSource-DiaObject join will be better informed once we can test the performance of that query on a prototype implementation. So for the purposes of the DPDD we will for now list the normalized version, sans ra/dec.
(Colin also pointed out a potential optimization: "One could also imagine making a field in DiaObject that records the most recent ccdVisitId, then one could do an equality join instead of the validity range.")
Adopted; implementation ticket is https://jira.lsstcorp.org/browse/DM-15677.
(comment deleted; mistakenly tagged this as relating to a different LCR)
Colin Slater
added a comment - - edited (comment deleted; mistakenly tagged this as relating to a different LCR)
As long as the fluxes are floats and not doubles, this should be consistent with the current sizing in LDM-141 and LDM-144 (although it is likely using up the "future growth" reserve for the DIAForcedSource table).