Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-517

Specify contents of DIAForcedSources in DPDD and include them in alerts

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:

      Description

      The DPDD describes (most completely in Section 3.2.1) the execution of forced and precovery photomery during prompt processing. Footnote 26 says that the DPDD treats them as "just" DIASources that may be flagged appropriately in the DIASource table or stored in a separate table. This statement creates some ambiguity I'd like to resolve.

      Because DIAForcedSources and DIASources have different semantic meanings, I'd first like to propose that we distinguish them explicitly in the DPDD (even if the actual storage of DIAForcedSources turns out to be as a flagged subset within the DIASource table). This would eliminate footnote 26, which is itself inconsistent with the statement in Section 3.3.1 that the DIASource table is "a table of sources detected at SNR > 5 on difference images." In my scan of the rest of the DPDD, actual usage of "DIASource" appears to be restricted to "unforced" DIASources, contra footnote 26.

      I propose that we further create a DIAForcedSources table that clarifies which columns will be present in that table. I believe these should be at least:
      diaForcedSourceId, ccdVisitId, diaObjectId, psFlux, psFluxErr (which is missing from DIASource!), flags.
      I also think we should include totFlux and totFluxErr (forced flux on the PVI): these are required to reconstruct the DC lightcurve of variable stars when the difference image magnitude is near that of the template.

      I also suggest we state explicitly that any existing DIAForcedSources already in the PPDB will be included in alert packets with the 12-month DIASource history. When available, these measurements are obviously more valuable to end-users than the upper limits provided by RFC-348.

      Finally, I suggest that we clarify that DIAObject timeseries features are computed on DIASources only and not DIAForcedSources (even if present), since the latter have complex sampling histories related to the precovery window and in many cases will be nondetections.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            As long as the fluxes are floats and not doubles, this should be consistent with the current sizing in LDM-141 and LDM-144 (although it is likely using up the "future growth" reserve for the DIAForcedSource table).

            Show
            ktl Kian-Tat Lim added a comment - As long as the fluxes are floats and not doubles, this should be consistent with the current sizing in LDM-141 and LDM-144 (although it is likely using up the "future growth" reserve for the DIAForcedSource table).
            Hide
            ctslater Colin Slater added a comment -

            I support this whole set of clarifications.

            The one other item I can think to add to the schema is the ra and dec that was input to the force photometry. Since the centroid of the DIAObject is updated each time a new DIASource is matched, the positions of a set of DIAForcedSource records won't necessarily be constant. I could imagine scenarios where one wants to know if variations in flux are due to that center wandering around. The position would otherwise need to be extracted from matching each DIAFS record to the specific DIAObject record that was valid at the time. (Though I can't say whether this is really worth the cost of two doubles x 10s of billions of records to avoid that complicated query)

            Show
            ctslater Colin Slater added a comment - I support this whole set of clarifications. The one other item I can think to add to the schema is the ra and dec that was input to the force photometry. Since the centroid of the DIAObject is updated each time a new DIASource is matched, the positions of a set of  DIAForcedSource records won't necessarily be constant. I could imagine scenarios where one wants to know if variations in flux are due to that center wandering around. The position would otherwise need to be extracted from matching each DIAFS record to the specific DIAObject record that was valid at the time. (Though I can't say whether this is really worth the cost of two doubles x 10s of billions of records to avoid that complicated query)
            Hide
            lguy Leanne Guy added a comment - - edited
            Show
            lguy Leanne Guy added a comment - - edited Created LCR-1459:  https://project.lsst.org/groups/ccb/node/2655
            Hide
            ebellm Eric Bellm added a comment -

            Colin Slater and I discussed the potential inclusion of the DiaObject ra and dec in the DiaForcedSourceTable. We agreed that the tradeoff between extra storage and performing a DiaForcedSource-DiaObject join will be better informed once we can test the performance of that query on a prototype implementation. So for the purposes of the DPDD we will for now list the normalized version, sans ra/dec.

            (Colin also pointed out a potential optimization: "One could also imagine making a field in DiaObject that records the most recent ccdVisitId, then one could do an equality join instead of the validity range.")

            Show
            ebellm Eric Bellm added a comment - Colin Slater and I discussed the potential inclusion of the DiaObject ra and dec in the DiaForcedSourceTable. We agreed that the tradeoff between extra storage and performing a DiaForcedSource-DiaObject join will be better informed once we can test the performance of that query on a prototype implementation. So for the purposes of the DPDD we will for now list the normalized version, sans ra/dec. (Colin also pointed out a potential optimization: "One could also imagine making a field in DiaObject that records the most recent ccdVisitId, then one could do an equality join instead of the validity range.")
            Hide
            ebellm Eric Bellm added a comment - - edited

            Adopted; implementation ticket is https://jira.lsstcorp.org/browse/DM-15677.

            Show
            ebellm Eric Bellm added a comment - - edited Adopted; implementation ticket is https://jira.lsstcorp.org/browse/DM-15677 .
            Hide
            ctslater Colin Slater added a comment - - edited

            (comment deleted; mistakenly tagged this as relating to a different LCR)

            Show
            ctslater Colin Slater added a comment - - edited (comment deleted; mistakenly tagged this as relating to a different LCR)

              People

              Assignee:
              ebellm Eric Bellm
              Reporter:
              ebellm Eric Bellm
              Watchers:
              Colin Slater, Eric Bellm, Gregory Dubois-Felsmann, Kian-Tat Lim, Leanne Guy
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins Builds

                  No builds found.